Nov 01 21:43:32.580301 ip-10-0-136-47 systemd[1]: Starting Kubernetes Kubelet... Nov 01 21:43:32.962498 ip-10-0-136-47 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:32.962498 ip-10-0-136-47 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 21:43:32.962498 ip-10-0-136-47 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:32.962498 ip-10-0-136-47 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:43:32.967884 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967812 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.967884 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967885 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967888 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967892 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967895 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967898 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967901 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967904 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967907 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967909 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967912 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967914 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967922 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967925 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967927 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967930 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967933 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967935 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967938 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967940 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.967950 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967943 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967946 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967948 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967951 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967954 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967956 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967959 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967962 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967966 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967971 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967974 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967977 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967980 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967983 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967985 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967988 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967990 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967993 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.967997 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.968407 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968000 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968003 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968005 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968008 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968010 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968013 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968016 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968018 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968020 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968036 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968039 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968042 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968044 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968047 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968050 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968053 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968056 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968059 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968061 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.968858 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968064 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968067 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968070 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968072 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968075 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968077 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968080 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968082 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968084 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968087 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968089 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968092 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968094 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968097 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968099 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968102 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968105 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968107 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968109 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.969310 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968112 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968115 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968117 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968120 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968122 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968124 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968127 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968130 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968132 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968136 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968138 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968141 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968143 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968146 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968148 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968151 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968153 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968155 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968158 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968160 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.969766 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968162 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968165 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968167 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968170 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968172 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968174 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968177 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968179 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.970241 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.968182 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.970555 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970542 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.970555 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970554 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970558 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970561 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970564 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970567 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970570 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970573 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970575 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970578 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970580 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970583 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970586 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970589 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970591 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970594 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970596 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970601 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970605 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.970608 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970609 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970613 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970616 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970621 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970624 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970627 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970629 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970631 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970634 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970637 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970639 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970641 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970644 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970646 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970649 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970651 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970653 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970656 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970658 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970661 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.971084 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970664 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970667 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970669 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970671 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970674 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970677 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970679 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970682 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970684 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970687 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970689 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970692 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970694 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970697 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970700 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970702 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970705 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970707 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970710 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970712 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.971557 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970715 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970717 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970720 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970722 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970725 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970727 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970729 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970732 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970734 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970737 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970739 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970742 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970745 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970747 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970750 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970752 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970755 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970758 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970760 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.972022 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970763 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970765 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970768 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970770 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970773 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970775 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970777 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970781 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970783 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970786 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970788 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970791 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970793 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970795 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970798 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970800 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970803 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970805 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970808 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970810 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.972479 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970813 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970816 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970818 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970821 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970823 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970825 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970828 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.970831 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971334 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971341 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971345 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971348 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971352 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971354 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971357 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971360 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971362 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971365 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971367 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.972945 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971370 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971373 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971376 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971378 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971381 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971383 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971386 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971388 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971391 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971394 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971397 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971399 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971402 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971405 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971407 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971410 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971412 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971414 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971418 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.973410 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971423 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971426 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971428 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971431 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971433 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971436 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971438 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971441 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971443 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971446 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971449 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971451 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971454 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971456 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971459 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971461 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971464 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971466 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971469 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971471 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.973871 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971474 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971478 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971481 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971483 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971486 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971488 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971490 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971493 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971495 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971498 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971500 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971504 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971507 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971510 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971513 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971515 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971518 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971520 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971523 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.974360 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971525 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971527 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971530 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971532 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971535 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971537 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971541 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971544 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971547 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971549 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971552 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971554 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971557 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971559 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971561 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971564 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971567 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971569 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971572 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.974825 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971574 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971576 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971579 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971581 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971584 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971587 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971589 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971592 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971595 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971598 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971601 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971603 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971606 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971608 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971611 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971613 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.971615 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.972678 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.972687 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973406 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973414 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 21:43:32.975287 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973418 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973422 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973427 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973431 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973434 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973437 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973441 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973444 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973447 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973450 2576 flags.go:64] FLAG: --cgroup-root="" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973453 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973456 2576 flags.go:64] FLAG: --client-ca-file="" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973459 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973461 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973465 2576 flags.go:64] FLAG: --cluster-domain="" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973468 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973472 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973475 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973478 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973482 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973485 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973488 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973491 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973494 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 01 21:43:32.975810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973497 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973500 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973503 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973505 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973509 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973512 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973515 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973518 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973521 2576 flags.go:64] FLAG: --enable-server="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973523 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973528 2576 flags.go:64] FLAG: --event-burst="100" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973531 2576 flags.go:64] FLAG: --event-qps="50" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973534 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973537 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973540 2576 flags.go:64] FLAG: --eviction-hard="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973544 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973547 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973550 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973553 2576 flags.go:64] FLAG: --eviction-soft="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973556 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973559 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973561 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973564 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973567 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973569 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 21:43:32.976397 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973572 2576 flags.go:64] FLAG: --feature-gates="" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973576 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973579 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973582 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973585 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973588 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973591 2576 flags.go:64] FLAG: --help="false" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973594 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973597 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973600 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973603 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973606 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973609 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973612 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973614 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973617 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973620 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973623 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973626 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973628 2576 flags.go:64] FLAG: --kube-reserved="" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973631 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973634 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973637 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 21:43:32.977007 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973640 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973642 2576 flags.go:64] FLAG: --lock-file="" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973645 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973648 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973659 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973664 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973667 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973670 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973673 2576 flags.go:64] FLAG: --logging-format="text" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973675 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973679 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973681 2576 flags.go:64] FLAG: --manifest-url="" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973684 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973689 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973692 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973695 2576 flags.go:64] FLAG: --max-pods="110" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973698 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973701 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973704 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973707 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973709 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973712 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973715 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973722 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973725 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 21:43:32.977569 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973728 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973731 2576 flags.go:64] FLAG: --pod-cidr="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973734 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973736 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973739 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973742 2576 flags.go:64] FLAG: --port="10250" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973745 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973748 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0cfe303a03f2d5efb" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973751 2576 flags.go:64] FLAG: --qos-reserved="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973753 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973756 2576 flags.go:64] FLAG: --register-node="true" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973759 2576 flags.go:64] FLAG: --register-with-taints="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973763 2576 flags.go:64] FLAG: --registry-burst="10" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973765 2576 flags.go:64] FLAG: --registry-qps="5" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973768 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973771 2576 flags.go:64] FLAG: --reserved-memory="" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973774 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973777 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973780 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973783 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973785 2576 flags.go:64] FLAG: --runonce="false" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973789 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973792 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973795 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973797 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973800 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 21:43:32.978195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973803 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973806 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973808 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973811 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973814 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973817 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973819 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973822 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973825 2576 flags.go:64] FLAG: --system-cgroups="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973827 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973832 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973835 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973841 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973845 2576 flags.go:64] FLAG: --tls-min-version="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973848 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973851 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973853 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973856 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973859 2576 flags.go:64] FLAG: --v="2" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973863 2576 flags.go:64] FLAG: --version="false" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973867 2576 flags.go:64] FLAG: --vmodule="" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973871 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.973874 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974010 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.978786 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974014 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974017 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974019 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974022 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974037 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974040 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974042 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974045 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974048 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974050 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974053 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974055 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974058 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974060 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974062 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974067 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974070 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974072 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.979397 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974075 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974080 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974083 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974085 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974088 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974091 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974093 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974096 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974099 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974101 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974104 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974106 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974109 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974111 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974114 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974116 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974119 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974121 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974124 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.979845 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974127 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974130 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974133 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974135 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974138 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974140 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974143 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974145 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974147 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974151 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974154 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974157 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974160 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974163 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974167 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974170 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974173 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974175 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974178 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974180 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.980370 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974183 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974185 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974188 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974190 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974193 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974195 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974198 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974200 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974203 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974205 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974208 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974210 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974213 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974216 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974219 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974221 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974224 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974226 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974228 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.980913 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974231 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974233 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974235 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974238 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974243 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974245 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974250 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974254 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974257 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974260 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974263 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974265 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974268 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974270 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974273 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974276 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974278 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974281 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974283 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.981631 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974286 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974289 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974291 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974293 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974296 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974299 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974302 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974304 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.974333 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974370 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974375 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974378 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974381 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974384 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974386 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974389 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.982401 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974392 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974394 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974396 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974400 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974404 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974407 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974409 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974412 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974414 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974417 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974419 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974422 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974424 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974428 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974432 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974434 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974437 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974440 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974443 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.982790 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974445 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974448 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974450 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974452 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974455 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974458 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974460 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974463 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974465 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974468 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974470 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974473 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974475 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974478 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974480 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974483 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974486 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974490 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974493 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.983260 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974496 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974498 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974501 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974504 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974506 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974509 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974511 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974514 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974516 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974518 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974521 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974524 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974527 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974530 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974534 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974537 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974539 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974542 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974545 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974548 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.983697 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974550 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974553 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974555 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974557 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974560 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974562 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974565 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974567 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974570 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974574 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974578 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974581 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974583 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974586 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974589 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974591 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974594 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974596 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974598 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.984244 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974601 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974603 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974606 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974609 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974613 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974616 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974618 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974621 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974623 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974626 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974628 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974631 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974634 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974636 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974640 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974642 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974645 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974647 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.984688 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974650 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974652 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.974655 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.983508 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.983525 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.984261 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.984268 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984331 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984335 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984339 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984343 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984346 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984349 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984351 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984354 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984358 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984360 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984363 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984365 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.985250 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984368 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984371 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984373 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984376 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984378 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984381 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984384 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984386 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984390 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984394 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984403 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984405 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984408 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984411 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984414 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984417 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984421 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984424 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984428 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.985699 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984430 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984433 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984435 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984438 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984441 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984443 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984445 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984448 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984450 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984452 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984455 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984457 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984460 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984462 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984465 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984467 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984470 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984472 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984475 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984477 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.986168 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984480 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984482 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984485 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984487 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984489 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984494 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984496 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984499 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984501 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984504 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984507 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984510 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984512 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984514 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984517 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984519 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984522 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984525 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984527 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.986627 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984529 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984532 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984534 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984536 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984539 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984541 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984543 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984546 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984549 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984551 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984554 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984556 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984558 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984561 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984563 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984565 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984568 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984570 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984572 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984575 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.987112 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984577 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984580 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984583 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984585 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984588 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984590 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984593 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984595 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984598 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984600 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984603 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984605 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984608 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984610 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984612 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.984640 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:32.987563 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984786 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984791 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984794 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984797 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984799 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984802 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984805 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984807 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984810 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984812 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984814 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984817 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984819 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984822 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984824 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984827 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984829 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984832 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984834 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.987925 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984837 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984840 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984842 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984844 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984847 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984850 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984852 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984855 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984857 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984860 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984862 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984865 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984867 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984869 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984872 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984874 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984877 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984879 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984882 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984886 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.988442 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984888 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984891 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984893 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984896 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984898 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984900 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984903 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984905 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984908 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984910 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984912 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984915 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984917 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984920 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984922 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984924 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984927 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984930 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984932 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984934 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.988903 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984936 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984939 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984941 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984944 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984946 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984948 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984951 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984955 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984959 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984962 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984965 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984968 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984973 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984976 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984979 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984981 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984984 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984986 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984989 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.989388 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984992 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984994 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984996 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.984999 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985002 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985004 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985006 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985009 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985011 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985014 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985016 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985018 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985021 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985040 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985045 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985048 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985050 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985053 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985055 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985058 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.989842 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985060 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985063 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985065 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985067 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985070 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985072 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985076 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985079 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.985103 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985134 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985138 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985142 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985146 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985149 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985152 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:43:32.990319 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985155 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985157 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985159 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985162 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985165 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985167 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985171 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985174 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985176 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985179 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985181 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985184 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985187 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985191 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985193 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985196 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985198 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985201 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985203 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985206 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:43:32.990664 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985208 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985211 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985213 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985216 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985219 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985221 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985224 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985226 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985229 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985231 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985234 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985236 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985239 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985241 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985244 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985246 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985249 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985251 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985255 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:43:32.991143 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985258 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985260 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985263 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985266 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985268 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985271 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985276 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985278 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985281 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985284 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985286 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985289 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985291 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985294 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985296 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985298 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985301 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985303 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985306 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:43:32.991615 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985309 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985311 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985314 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985316 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985319 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985321 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985323 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985326 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985328 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985331 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985333 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985336 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985340 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985343 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985345 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985348 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985350 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985352 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985355 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985359 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:43:32.992079 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985361 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985364 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985366 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985369 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985371 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985374 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985376 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985378 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985381 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985383 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985386 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985388 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985391 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985393 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985396 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985398 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985401 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985403 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985406 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:43:32.992562 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985408 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985412 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:32.985416 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.985470 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.987821 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.988859 2576 server.go:1014] "Starting client certificate rotation" Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.988953 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.988987 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:43:32.993003 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.991919 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:43:32.995064 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:32.995045 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:43:33.006127 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.006110 2576 log.go:25] "Validated CRI v1 runtime API" Nov 01 21:43:33.010041 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.010013 2576 log.go:25] "Validated CRI v1 image API" Nov 01 21:43:33.010893 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.010873 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 21:43:33.013595 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.013578 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d76ff796-256e-467c-9aa4-af9533a85cd9:/dev/nvme0n1p4 dbb5e605-2cc4-4937-8e3a-42faa5b4001b:/dev/nvme0n1p3] Nov 01 21:43:33.013662 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.013596 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 21:43:33.019753 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.019639 2576 manager.go:217] Machine: {Timestamp:2026-11-01 21:43:33.018137901 +0000 UTC m=+0.344156926 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095316 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2778a09e19b7ac909172f8ad5b4549 SystemUUID:ec2778a0-9e19-b7ac-9091-72f8ad5b4549 BootID:64564fa0-f11e-4a12-8339-2bc6743cad34 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:6f:52:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:6f:52:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:b3:99:00:19:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 21:43:33.019753 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.019748 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 21:43:33.019874 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.019856 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 21:43:33.022371 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.022355 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 21:43:33.023375 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023353 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 21:43:33.023502 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023379 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-47.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 21:43:33.023542 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023504 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 21:43:33.023542 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023528 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 21:43:33.023593 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023567 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:43:33.023593 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.023580 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:43:33.024854 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.024844 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 21:43:33.029364 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029353 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 21:43:33.029467 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029459 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 21:43:33.029500 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029489 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 01 21:43:33.029500 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029498 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 21:43:33.029548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029508 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 21:43:33.029548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029517 2576 kubelet.go:405] "Adding apiserver pod source" Nov 01 21:43:33.029548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029525 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 21:43:33.029830 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.029816 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:43:33.032304 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.032286 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 21:43:33.038973 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.038959 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:43:33.039940 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.039922 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 21:43:33.040046 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.039962 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 21:43:33.040174 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040161 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 21:43:33.040233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040181 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 21:43:33.040233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040192 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 21:43:33.040233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040202 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 21:43:33.040233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040210 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 21:43:33.040233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040219 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 21:43:33.040960 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040950 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 21:43:33.041013 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040964 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 21:43:33.041013 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040974 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 21:43:33.041013 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.040986 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 21:43:33.041013 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.041010 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 21:43:33.041224 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.041044 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 21:43:33.041824 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.041809 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 21:43:33.041881 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.041826 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 21:43:33.042785 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.042762 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vjnlw" Nov 01 21:43:33.045170 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.045148 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 21:43:33.045331 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.045320 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-47.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 21:43:33.045471 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.045453 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-47.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 21:43:33.045743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.045729 2576 server.go:1263] "Started kubelet" Nov 01 21:43:33.045814 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.045788 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 21:43:33.046195 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.046160 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 21:43:33.046268 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.046216 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 21:43:33.046432 ip-10-0-136-47 systemd[1]: Started Kubernetes Kubelet. Nov 01 21:43:33.047635 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.047617 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 21:43:33.048727 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.048713 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 01 21:43:33.052556 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.052529 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 21:43:33.054115 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.054099 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vjnlw" Nov 01 21:43:33.061974 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.061957 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 21:43:33.062581 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.062555 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 21:43:33.063198 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.062678 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.063360 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063344 2576 factory.go:55] Registering systemd factory Nov 01 21:43:33.063433 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063363 2576 factory.go:223] Registration of the systemd container factory successfully Nov 01 21:43:33.063433 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063364 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 21:43:33.063433 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063378 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 21:43:33.063582 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063444 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 21:43:33.063582 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063481 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 01 21:43:33.063582 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063487 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 01 21:43:33.063582 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063545 2576 factory.go:153] Registering CRI-O factory Nov 01 21:43:33.063582 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063554 2576 factory.go:223] Registration of the crio container factory successfully Nov 01 21:43:33.063801 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063595 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 21:43:33.063801 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063620 2576 factory.go:103] Registering Raw factory Nov 01 21:43:33.063801 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.063632 2576 manager.go:1199] Started watching for new ooms in manager Nov 01 21:43:33.064614 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.064598 2576 manager.go:319] Starting recovery of all containers Nov 01 21:43:33.065394 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.061630 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-47.us-east-2.compute.internal.18e409e0e098583c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-47.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-47.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 21:43:33.045704764 +0000 UTC m=+0.371723789,LastTimestamp:2026-11-01 21:43:33.045704764 +0000 UTC m=+0.371723789,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-47.us-east-2.compute.internal,}" Nov 01 21:43:33.075450 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.075434 2576 manager.go:324] Recovery completed Nov 01 21:43:33.077858 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.077839 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:33.079505 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.079490 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.080503 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.080483 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-47.us-east-2.compute.internal\" not found" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.081794 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.081776 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.081884 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.081813 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.081884 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.081829 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.082345 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.082334 2576 cpu_manager.go:226] "Starting" policy="none" Nov 01 21:43:33.082345 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.082345 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 21:43:33.082418 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.082359 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 21:43:33.082418 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.082392 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.083070 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.083055 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.083133 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.083080 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.083133 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.083091 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.084522 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.084510 2576 policy_none.go:50] "Start" Nov 01 21:43:33.084554 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.084526 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 21:43:33.084554 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.084538 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 21:43:33.086528 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.086519 2576 policy_none.go:44] "Start" Nov 01 21:43:33.124654 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.124534 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 21:43:33.128454 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128442 2576 manager.go:341] "Starting Device Plugin manager" Nov 01 21:43:33.128514 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.128469 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 21:43:33.128514 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128477 2576 server.go:87] "Starting device plugin registration server" Nov 01 21:43:33.128702 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128691 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 21:43:33.128744 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128704 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 21:43:33.128800 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128783 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 21:43:33.128894 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128878 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 21:43:33.128894 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.128892 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 21:43:33.129350 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.129332 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 21:43:33.129418 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.129381 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.192245 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.192225 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 21:43:33.194233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.193473 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 21:43:33.194233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.193495 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 21:43:33.194233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.193513 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 21:43:33.194233 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.193541 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 21:43:33.197229 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.197211 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:33.229656 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.229612 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.230366 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.230347 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.230442 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.230379 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.230442 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.230394 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.230442 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.230416 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.238763 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.238749 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.238805 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.238769 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-47.us-east-2.compute.internal\": node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.254869 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.254846 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.295223 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.294968 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal"] Nov 01 21:43:33.295906 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.295694 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.297797 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.297782 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.297856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.297812 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.297856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.297828 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.300160 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300149 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.300311 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.300349 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300327 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.300791 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300777 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.300856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300794 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.300856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300805 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.300856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300814 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.300856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300820 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.300856 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.300824 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.302962 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.302943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.303049 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.302978 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:43:33.303673 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.303657 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:43:33.303718 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.303691 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:43:33.303718 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.303705 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:43:33.322420 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.322406 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.324530 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.324514 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.355196 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.355178 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.364725 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.364705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.364780 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.364735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.364780 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.364754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.455806 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.455772 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.464971 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.464950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.465059 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.464975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.465059 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.464991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.465059 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.465016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.465059 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.465059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.465207 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.465076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.556390 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.556359 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.625839 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.625814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.627934 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.627916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:33.656451 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.656433 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.756855 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.756830 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.857328 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.857284 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.936754 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.936733 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:33.957678 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:33.957648 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:33.989098 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.989076 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 21:43:33.989471 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.989173 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 21:43:33.989471 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.989210 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 21:43:33.989471 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:33.989232 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 21:43:34.057977 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:34.057955 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:34.061139 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.061104 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:33 +0000 UTC" deadline="2028-05-01 10:22:31.388175288 +0000 UTC" Nov 01 21:43:34.061139 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.061139 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13116h38m57.327038619s" Nov 01 21:43:34.158991 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:34.158962 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:34.245951 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:34.245921 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04a04cb275f35547de27e44efd5af4f.slice/crio-c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b WatchSource:0}: Error finding container c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b: Status 404 returned error can't find the container with id c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b Nov 01 21:43:34.246390 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:34.246370 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91729725e20792f4990b48a3f50d1312.slice/crio-6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499 WatchSource:0}: Error finding container 6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499: Status 404 returned error can't find the container with id 6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499 Nov 01 21:43:34.250687 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.250666 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:43:34.259448 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:34.259430 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:34.359904 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:34.359884 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:43:34.408461 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.408417 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:34.463494 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.463476 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:34.475910 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.475896 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:43:34.476707 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.476696 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:43:34.486132 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:34.486117 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:43:35.031421 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.031164 2576 apiserver.go:52] "Watching apiserver" Nov 01 21:43:35.045906 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.045857 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 21:43:35.046310 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.046275 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-z8p6m","kube-system/konnectivity-agent-7wgdj","kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg","openshift-dns/node-resolver-mc686","openshift-image-registry/node-ca-gbcqn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-qs8p9","kube-system/global-pull-secret-syncer-4w9sx","openshift-multus/multus-additional-cni-plugins-6rxgf","openshift-cluster-node-tuning-operator/tuned-nf8vf","openshift-multus/multus-q67sd","openshift-network-diagnostics/network-check-target-qghnn","openshift-network-operator/iptables-alerter-c2ntz"] Nov 01 21:43:35.048946 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.048925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.051322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.051299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:35.051459 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.051433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:35.053608 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.053586 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.053781 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.053756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.054017 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.053759 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 21:43:35.056270 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.056248 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 21:43:35.056362 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.056281 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 21:43:35.056362 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.056306 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.056488 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.056374 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\"" Nov 01 21:43:35.056922 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.056903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.060237 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.059556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.063196 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.063177 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 21:43:35.063575 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.063545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.070347 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.070325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.070444 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.070423 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:35.070758 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.070600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.070758 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.070717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.072355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trdc\" (UniqueName: \"kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfqs\" (UniqueName: \"kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8vx\" (UniqueName: \"kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwrg\" (UniqueName: \"kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.074987 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.073734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.077694 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.077674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.078707 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.078684 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 21:43:35.079718 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.079696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.081814 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.081787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.096133 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.096111 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8hn9m" Nov 01 21:43:35.113969 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.113949 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8hn9m" Nov 01 21:43:35.162892 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.162867 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 21:43:35.162982 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.162936 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.163270 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.163249 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.163372 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.163405 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.163683 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164216 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164264 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164396 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 21:43:35.164459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164405 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 21:43:35.164760 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164493 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\"" Nov 01 21:43:35.164760 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164722 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.164760 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164758 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.164849 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164810 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.164849 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164841 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 21:43:35.164927 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164892 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\"" Nov 01 21:43:35.165073 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.164993 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.165073 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.165046 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.165073 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.165059 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.165869 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.165849 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.165975 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.165962 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:43:35.166340 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.166239 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 21:43:35.166340 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.166293 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\"" Nov 01 21:43:35.166340 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.166283 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 21:43:35.166524 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.166410 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 21:43:35.167891 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.167872 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\"" Nov 01 21:43:35.167982 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.167896 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\"" Nov 01 21:43:35.167982 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.167934 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\"" Nov 01 21:43:35.168121 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.167989 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 21:43:35.168121 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.168007 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\"" Nov 01 21:43:35.168333 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.168314 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 21:43:35.173911 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.173889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.174011 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.173924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.174011 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.173986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174124 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.174124 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:35.174194 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174194 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.174194 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174194 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174313 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.174313 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.174377 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174321 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174377 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.174464 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.174530 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.174530 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8vx\" (UniqueName: \"kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.174530 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.174693 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.174693 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.174693 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.174693 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.174693 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.174950 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.174684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.175082 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.175051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.175332 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.175306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.175903 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.175884 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.175996 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.175969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.176148 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.175361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.176361 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.176341 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.180892 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.180861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.180990 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.180935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sww5\" (UniqueName: \"kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.181137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.181261 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181236 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.181326 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.181326 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.181407 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.181459 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.181498 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.181532 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h25l\" (UniqueName: \"kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.181635 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.181723 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.181723 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.181691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.182309 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182288 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.182380 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.182514 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.182553 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.182789 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.182839 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.182873 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trdc\" (UniqueName: \"kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.182906 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.182957 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.183009 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.182994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.183116 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.183176 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.183233 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.183307 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183457 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183803 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183872 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.183872 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfqs\" (UniqueName: \"kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183952 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.183978 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.183964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwrg\" (UniqueName: \"kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z9fn\" (UniqueName: \"kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.184688 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184700 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.184876 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.184950 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:35.684927834 +0000 UTC m=+3.010946864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:35.185253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.184957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.185877 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.185858 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:35.185936 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.185884 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:35.185936 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.185899 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:35.186048 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.185948 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:35.68593221 +0000 UTC m=+3.011951242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:35.186048 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.185977 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 21:43:35.190202 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.190171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8vx\" (UniqueName: \"kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.196895 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.196868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trdc\" (UniqueName: \"kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.197011 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.196873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.199954 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.199922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfqs\" (UniqueName: \"kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.200076 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.199977 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwrg\" (UniqueName: \"kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.202114 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.202043 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"91729725e20792f4990b48a3f50d1312","Type":"ContainerStarted","Data":"6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499"} Nov 01 21:43:35.203414 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.203369 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerStarted","Data":"c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b"} Nov 01 21:43:35.239343 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.239320 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:35.285548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285714 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285711 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.285743 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285660 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285588 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z9fn\" (UniqueName: \"kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.285994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286170 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.286277 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286186 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286229 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286389 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.287137 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286600 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sww5\" (UniqueName: \"kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.286798 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286846 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.286859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:43:35.7868395 +0000 UTC m=+3.112858535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h25l\" (UniqueName: \"kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286976 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.288038 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.286933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287388 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287389 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.289146 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287507 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287641 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287626 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287823 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.287866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290095 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.288292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.290700 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.290347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290700 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.290378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.290700 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.290472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.290700 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.290518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.300523 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.300501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z9fn\" (UniqueName: \"kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.301035 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.300986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.301439 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.301418 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h25l\" (UniqueName: \"kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.301564 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.301545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sww5\" (UniqueName: \"kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.345492 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.345467 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:35.383264 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.383239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:43:35.385430 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.385404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:43:35.390876 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.390853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:43:35.393359 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.393326 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1177d05_06de_4967_b190_b00451f8f853.slice/crio-ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2 WatchSource:0}: Error finding container ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2: Status 404 returned error can't find the container with id ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2 Nov 01 21:43:35.397170 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.397075 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b809a87_86be_4c57_a0e5_e491568d34c1.slice/crio-708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e WatchSource:0}: Error finding container 708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e: Status 404 returned error can't find the container with id 708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e Nov 01 21:43:35.402013 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.401991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:43:35.409163 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.409138 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea9a939_d8f8_4d26_b7eb_57e2b88ba5c5.slice/crio-0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87 WatchSource:0}: Error finding container 0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87: Status 404 returned error can't find the container with id 0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87 Nov 01 21:43:35.410893 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.410864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mc686" Nov 01 21:43:35.418240 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.418219 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe1ba52_ea23_43ce_9286_975b58f3998d.slice/crio-66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314 WatchSource:0}: Error finding container 66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314: Status 404 returned error can't find the container with id 66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314 Nov 01 21:43:35.419357 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.419339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:35.423526 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.423029 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2b9e93_478d_4b5f_bb3d_301f8b0c8be0.slice/crio-b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813 WatchSource:0}: Error finding container b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813: Status 404 returned error can't find the container with id b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813 Nov 01 21:43:35.434404 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.434383 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff79f3e_c818_4103_b626_3bed7d7b46b4.slice/crio-f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a WatchSource:0}: Error finding container f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a: Status 404 returned error can't find the container with id f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a Nov 01 21:43:35.442929 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.442907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:43:35.445369 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.445348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q67sd" Nov 01 21:43:35.449373 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.449353 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:35.455094 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.455072 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0464581_54a9_4673_a69c_3b560089a3c4.slice/crio-f33cb4df203e0e374c8353957f522c358e695ccb698b5f567155ff3d7af62190 WatchSource:0}: Error finding container f33cb4df203e0e374c8353957f522c358e695ccb698b5f567155ff3d7af62190: Status 404 returned error can't find the container with id f33cb4df203e0e374c8353957f522c358e695ccb698b5f567155ff3d7af62190 Nov 01 21:43:35.461209 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.461179 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded118150_74ab_4355_8f26_22d188e0140b.slice/crio-8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462 WatchSource:0}: Error finding container 8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462: Status 404 returned error can't find the container with id 8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462 Nov 01 21:43:35.462211 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:43:35.462186 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f71dcf_17d5_4fb3_ad84_c83bf8fb40ac.slice/crio-dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c WatchSource:0}: Error finding container dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c: Status 404 returned error can't find the container with id dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c Nov 01 21:43:35.691439 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.691344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:35.691439 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.691416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691513 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691564 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691607 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691618 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691588 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:36.691567672 +0000 UTC m=+4.017586702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:35.691669 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.691670 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:36.691653417 +0000 UTC m=+4.017672434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:35.792745 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.792712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:35.792894 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.792828 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:35.792894 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:35.792880 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:43:36.79286717 +0000 UTC m=+4.118886198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:35.806741 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:35.806716 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:43:36.118300 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.118176 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:35 +0000 UTC" deadline="2028-05-21 17:45:37.595956594 +0000 UTC" Nov 01 21:43:36.118300 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.118235 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13604h2m1.477725318s" Nov 01 21:43:36.214887 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.214742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q67sd" event={"ID":"ed118150-74ab-4355-8f26-22d188e0140b","Type":"ContainerStarted","Data":"8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462"} Nov 01 21:43:36.217850 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.217781 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wgdj" event={"ID":"5ff79f3e-c818-4103-b626-3bed7d7b46b4","Type":"ContainerStarted","Data":"f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a"} Nov 01 21:43:36.227518 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.227489 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mc686" event={"ID":"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0","Type":"ContainerStarted","Data":"b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813"} Nov 01 21:43:36.236735 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.236707 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" event={"ID":"0fe1ba52-ea23-43ce-9286-975b58f3998d","Type":"ContainerStarted","Data":"66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314"} Nov 01 21:43:36.257144 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.257115 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbcqn" event={"ID":"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5","Type":"ContainerStarted","Data":"0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87"} Nov 01 21:43:36.268006 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.267976 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerStarted","Data":"708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e"} Nov 01 21:43:36.280081 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.280055 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2ntz" event={"ID":"b1177d05-06de-4967-b190-b00451f8f853","Type":"ContainerStarted","Data":"ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2"} Nov 01 21:43:36.283518 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.283473 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c"} Nov 01 21:43:36.307753 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.306731 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" event={"ID":"d0464581-54a9-4673-a69c-3b560089a3c4","Type":"ContainerStarted","Data":"f33cb4df203e0e374c8353957f522c358e695ccb698b5f567155ff3d7af62190"} Nov 01 21:43:36.700371 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.700142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:36.700495 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.700440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:36.700631 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.700612 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:36.700695 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.700642 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:36.700695 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.700655 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:36.700842 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.700713 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:38.700689455 +0000 UTC m=+6.026708476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:36.701009 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.700990 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:36.701139 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.701048 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:38.701032937 +0000 UTC m=+6.027051965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:36.801281 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:36.801249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:36.801424 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.801399 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:36.801486 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:36.801462 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:43:38.801440797 +0000 UTC m=+6.127459826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:37.119435 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.119365 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:35 +0000 UTC" deadline="2028-05-18 14:10:08.285238019 +0000 UTC" Nov 01 21:43:37.119435 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.119403 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13528h26m31.165840707s" Nov 01 21:43:37.196144 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.195199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:37.196144 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:37.195444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:37.196144 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.195918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:37.196144 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:37.196032 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:37.196144 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.196094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:37.198280 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:37.198231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:37.331267 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.330783 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"91729725e20792f4990b48a3f50d1312","Type":"ContainerStarted","Data":"acaa439727a289502928e2c7305c4169b20d51f6bde5f586fe4c51a8442d4afe"} Nov 01 21:43:37.335723 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.335688 2576 generic.go:356] "Generic (PLEG): container finished" podID="f04a04cb275f35547de27e44efd5af4f" containerID="d7e7f3bd94257ebe4930530d6258cbde237424c345ee9d8cf79568ffb9c8f928" exitCode=0 Nov 01 21:43:37.335837 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.335742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerDied","Data":"d7e7f3bd94257ebe4930530d6258cbde237424c345ee9d8cf79568ffb9c8f928"} Nov 01 21:43:37.348345 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:37.348289 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" podStartSLOduration=2.348272981 podStartE2EDuration="2.348272981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:37.347346072 +0000 UTC m=+4.673365112" watchObservedRunningTime="2026-11-01 21:43:37.348272981 +0000 UTC m=+4.674292023" Nov 01 21:43:38.359986 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:38.359815 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerStarted","Data":"070b0c1ba29bbefda936de123daf34f3246171324456ac0af4654d2f1f3f9ba9"} Nov 01 21:43:38.724111 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:38.724071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:38.724289 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:38.724175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:38.724377 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724307 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:38.724447 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724375 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.724354907 +0000 UTC m=+10.050373936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:38.724833 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724810 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:38.724922 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724838 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:38.724922 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724852 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:38.724922 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.724899 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.724884426 +0000 UTC m=+10.050903445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:38.824753 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:38.824676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:38.824997 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.824830 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:38.824997 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:38.824896 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:43:42.824876707 +0000 UTC m=+10.150895743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:39.194206 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.194064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:39.194409 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:39.194212 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:39.194681 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.194643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:39.194804 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:39.194769 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:39.194875 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.194845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:39.194960 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:39.194927 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:39.368286 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.366835 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="3c2452ff61ff172b0d82faa54a97e8754aecb89483f0b616c8ea46e7fa7cc9cb" exitCode=0 Nov 01 21:43:39.368286 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.367974 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"3c2452ff61ff172b0d82faa54a97e8754aecb89483f0b616c8ea46e7fa7cc9cb"} Nov 01 21:43:39.397032 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:39.396436 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" podStartSLOduration=4.396415769 podStartE2EDuration="4.396415769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:38.379026373 +0000 UTC m=+5.705045411" watchObservedRunningTime="2026-11-01 21:43:39.396415769 +0000 UTC m=+6.722434807" Nov 01 21:43:40.372301 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:40.372259 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7wgdj" event={"ID":"5ff79f3e-c818-4103-b626-3bed7d7b46b4","Type":"ContainerStarted","Data":"3cd3cf4c5c1501a8804920e9e2782c86940f0109215a2d9d628d7d7d44c09ede"} Nov 01 21:43:40.606250 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:40.606098 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:40.607112 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:40.606892 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:40.625910 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:40.625550 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7wgdj" podStartSLOduration=2.463209199 podStartE2EDuration="6.625529574s" totalImagesPullingTime="4.162320375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:40.391748395 +0000 UTC m=+7.717767435" watchObservedRunningTime="2026-11-01 21:43:40.625529574 +0000 UTC m=+7.951548612" Nov 01 21:43:41.195058 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:41.194256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:41.195058 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:41.194434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:41.195058 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:41.194865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:41.195058 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:41.195005 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:41.195461 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:41.195427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:41.203510 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:41.199259 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:41.379583 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:41.379543 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbcqn" event={"ID":"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5","Type":"ContainerStarted","Data":"9f406a812ff5dacf6e06d0be95dd8dec54d4f38bd966b8f817c39c1898222817"} Nov 01 21:43:41.397823 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:41.397739 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gbcqn" podStartSLOduration=1.8985440900000001 podStartE2EDuration="7.39771697s" totalImagesPullingTime="5.49917288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:41.396763888 +0000 UTC m=+8.722782927" watchObservedRunningTime="2026-11-01 21:43:41.39771697 +0000 UTC m=+8.723736010" Nov 01 21:43:42.384477 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:42.384441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" event={"ID":"0fe1ba52-ea23-43ce-9286-975b58f3998d","Type":"ContainerStarted","Data":"886cc49be1fa060f744a05dda48f4aaba2311817404a9dca842960d4ca7f6d16"} Nov 01 21:43:42.384964 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:42.384559 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:42.763460 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:42.763420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:42.763654 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:42.763475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:42.763654 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.763595 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:42.763654 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.763610 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:42.763654 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.763619 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:42.763883 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.763667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:50.763654145 +0000 UTC m=+18.089673162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:42.764011 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.763991 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:42.764091 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.764047 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:50.764030255 +0000 UTC m=+18.090049274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:42.864841 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:42.864676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:42.864841 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.864832 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:42.865060 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:42.864902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:43:50.864880615 +0000 UTC m=+18.190899634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:43.195810 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:43.195732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:43.195944 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:43.195870 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:43.196291 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:43.196245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:43.196433 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:43.196388 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:43.196499 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:43.196459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:43.196571 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:43.196553 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:43.589711 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:43.589679 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 21:43:44.140036 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.139931 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:43:43.589706325Z","UUID":"4100c379-f8c8-4b6b-abc5-3bd505676fc0","Handler":null,"Name":"","Endpoint":""} Nov 01 21:43:44.141829 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.141806 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 21:43:44.141935 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.141840 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 21:43:44.394432 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.394186 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q67sd" event={"ID":"ed118150-74ab-4355-8f26-22d188e0140b","Type":"ContainerStarted","Data":"1eb2b03994696a0fbbb013b206bdb13e8c63cd6bfe7fa6cedb4a9d9d1fd67675"} Nov 01 21:43:44.398557 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.398485 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mc686" event={"ID":"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0","Type":"ContainerStarted","Data":"6c30e3ca0d579c52557ab915e6d70013eae13c852103573033aa383a712528f3"} Nov 01 21:43:44.403526 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.403497 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" event={"ID":"0fe1ba52-ea23-43ce-9286-975b58f3998d","Type":"ContainerStarted","Data":"2559c9b25a62b9383a328d63f8b67002511628273860d8784f6216607b833b04"} Nov 01 21:43:44.406828 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.406792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerStarted","Data":"e261e5777668766d9eca6bd99a089ffc6d842bb6ea576de41a33e950ba422364"} Nov 01 21:43:44.421824 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.421767 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q67sd" podStartSLOduration=2.173574758 podStartE2EDuration="10.421745973s" totalImagesPullingTime="8.248171215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:44.418873973 +0000 UTC m=+11.744893020" watchObservedRunningTime="2026-11-01 21:43:44.421745973 +0000 UTC m=+11.747765003" Nov 01 21:43:44.464667 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:44.464610 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mc686" podStartSLOduration=1.9428369189999999 podStartE2EDuration="10.464592929s" totalImagesPullingTime="8.52175601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:44.463831494 +0000 UTC m=+11.789850534" watchObservedRunningTime="2026-11-01 21:43:44.464592929 +0000 UTC m=+11.790611967" Nov 01 21:43:45.194085 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.193991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:45.194085 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.194074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:45.194085 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.194075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:45.194734 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:45.194172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:45.194734 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:45.194521 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:45.194734 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:45.194660 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:45.412240 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.412201 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" event={"ID":"0fe1ba52-ea23-43ce-9286-975b58f3998d","Type":"ContainerStarted","Data":"241299c6a6653082759887e8c910e544639af305aba350267b2a823e50fd1e8f"} Nov 01 21:43:45.414752 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.414722 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="e261e5777668766d9eca6bd99a089ffc6d842bb6ea576de41a33e950ba422364" exitCode=0 Nov 01 21:43:45.414885 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.414760 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"e261e5777668766d9eca6bd99a089ffc6d842bb6ea576de41a33e950ba422364"} Nov 01 21:43:45.431847 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:45.431798 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" podStartSLOduration=2.529017047 podStartE2EDuration="11.431783282s" totalImagesPullingTime="8.902766235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:45.431408868 +0000 UTC m=+12.757427903" watchObservedRunningTime="2026-11-01 21:43:45.431783282 +0000 UTC m=+12.757802318" Nov 01 21:43:46.418495 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:46.418456 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2ntz" event={"ID":"b1177d05-06de-4967-b190-b00451f8f853","Type":"ContainerStarted","Data":"2adcc684e427af291f84f996fb04f04976c02828403e8d7747ce99960c05b2b4"} Nov 01 21:43:46.435176 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:46.435127 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c2ntz" podStartSLOduration=3.707808185 podStartE2EDuration="12.435114266s" totalImagesPullingTime="8.727306081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:46.434477122 +0000 UTC m=+13.760496155" watchObservedRunningTime="2026-11-01 21:43:46.435114266 +0000 UTC m=+13.761133300" Nov 01 21:43:47.194548 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.194510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:47.194717 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.194543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:47.194717 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.194632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:47.194717 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:47.194648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:47.194842 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:47.194760 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:47.194892 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:47.194872 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:47.423801 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.423770 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="972bdc355742a902915e25ea112ee2faf5a8c72f453d89174fc5ee4d5b2cf5ee" exitCode=0 Nov 01 21:43:47.424587 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.423842 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"972bdc355742a902915e25ea112ee2faf5a8c72f453d89174fc5ee4d5b2cf5ee"} Nov 01 21:43:47.425592 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.425563 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" event={"ID":"d0464581-54a9-4673-a69c-3b560089a3c4","Type":"ContainerStarted","Data":"a38ac8f5a735422af506ed23aaa554b6605d29b4ff1bce637582866ef3dcb4ab"} Nov 01 21:43:47.469068 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:47.468978 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" podStartSLOduration=2.231940415 podStartE2EDuration="13.468965153s" totalImagesPullingTime="11.237024738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:47.468649211 +0000 UTC m=+14.794668246" watchObservedRunningTime="2026-11-01 21:43:47.468965153 +0000 UTC m=+14.794984188" Nov 01 21:43:48.434168 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:48.430213 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerStarted","Data":"818c1193dadbc2dea8ecf16cde9747877dd6d25aefb30eced3d7f965d9f83385"} Nov 01 21:43:49.194850 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:49.194808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:49.195010 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:49.194878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:49.195010 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:49.194873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:49.195010 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:49.194976 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:49.195299 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:49.195257 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:49.195387 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:49.195348 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:49.435591 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:49.435553 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="818c1193dadbc2dea8ecf16cde9747877dd6d25aefb30eced3d7f965d9f83385" exitCode=0 Nov 01 21:43:49.436081 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:49.435608 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"818c1193dadbc2dea8ecf16cde9747877dd6d25aefb30eced3d7f965d9f83385"} Nov 01 21:43:50.825773 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:50.825732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:50.825805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825892 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825900 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825915 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825927 2576 projected.go:196] Error preparing data for projected volume kube-api-access-v6d62 for pod openshift-network-diagnostics/network-check-target-qghnn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825951 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret podName:6eaffc4a-a597-4820-8a11-31e92606f9f2 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:06.825935952 +0000 UTC m=+34.151954980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret") pod "global-pull-secret-syncer-4w9sx" (UID: "6eaffc4a-a597-4820-8a11-31e92606f9f2") : object "kube-system"/"original-pull-secret" not registered Nov 01 21:43:50.826241 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.825967 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62 podName:a0158144-5a21-4098-869a-dd1598f3aad8 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:06.825958919 +0000 UTC m=+34.151977935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6d62" (UniqueName: "kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62") pod "network-check-target-qghnn" (UID: "a0158144-5a21-4098-869a-dd1598f3aad8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 21:43:50.926586 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:50.926431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:50.926586 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.926553 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:50.926718 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:50.926610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs podName:c83a4e94-9b7d-435a-bd02-b043476da87a nodeName:}" failed. No retries permitted until 2026-11-01 21:44:06.926592228 +0000 UTC m=+34.252611263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs") pod "network-metrics-daemon-qs8p9" (UID: "c83a4e94-9b7d-435a-bd02-b043476da87a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 21:43:51.194861 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.194681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:51.194951 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.194710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:51.195002 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.194755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:51.195077 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:51.195046 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:51.195136 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:51.195114 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:51.195184 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:51.194981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:51.442936 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.442855 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"02b87c7421295adb93c11c329890b24bda9d394701eb579881dc9dfe757db0b0"} Nov 01 21:43:51.443109 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.442940 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"e1c50e7a89b266c1e4e7bdce7b9461611836835a5d7b78b3c950a45bde947773"} Nov 01 21:43:51.443109 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.442984 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"d278797b3a64790df9875deddc658bc40c0804958e4a3f3305f3f680ae22e7de"} Nov 01 21:43:51.443109 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.443047 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"1496417bf8e5454d13ef471366f5abe15d4cbd45b8ac052c98c035c122aae5de"} Nov 01 21:43:51.443109 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.443092 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"b5a5cf1f5353fb960f30b128e6a4a064aedb954c4047de813fc3a8391a16ddc2"} Nov 01 21:43:51.443253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:51.443135 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"f1174a84bf95c679948503b3b7a93751c12ea83a6dd494fd4c082320028a9d6a"} Nov 01 21:43:52.447419 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:52.447390 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="a4411d0b19fece45da98305dae575a65e95da0d07b9e271c57589ff12ee1f716" exitCode=0 Nov 01 21:43:52.447787 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:52.447441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"a4411d0b19fece45da98305dae575a65e95da0d07b9e271c57589ff12ee1f716"} Nov 01 21:43:53.195344 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.195314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:53.195495 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:53.195404 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:53.196108 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.195771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:53.196108 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:53.195910 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:53.196108 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.195941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:53.196324 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:53.196155 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:53.451970 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.451899 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="d4464094dfd9cce4407b055296e69df5502d1255121722e12b890b1c910960f2" exitCode=0 Nov 01 21:43:53.452591 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.451977 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerDied","Data":"d4464094dfd9cce4407b055296e69df5502d1255121722e12b890b1c910960f2"} Nov 01 21:43:53.455642 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:53.455618 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"8028869baf23c000149ac8d0f54319d11eaaff7b7d60e6ed7e989d949bb58c8d"} Nov 01 21:43:54.460850 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:54.460816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" event={"ID":"3b809a87-86be-4c57-a0e5-e491568d34c1","Type":"ContainerStarted","Data":"6edbc613cb08015a3ea68a69bb86fb2bd18fa582002fc44c50ba460bc225a4d4"} Nov 01 21:43:54.487976 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:54.487932 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6rxgf" podStartSLOduration=8.622378002 podStartE2EDuration="20.487919272s" totalImagesPullingTime="11.86554127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:54.48789133 +0000 UTC m=+21.813910364" watchObservedRunningTime="2026-11-01 21:43:54.487919272 +0000 UTC m=+21.813938299" Nov 01 21:43:55.193942 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:55.193914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:55.193942 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:55.193946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:55.194212 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:55.194020 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:55.194212 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:55.194127 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:55.194212 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:55.194200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:55.194305 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:55.194290 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:55.467161 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:55.467080 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" event={"ID":"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac","Type":"ContainerStarted","Data":"4c460c57deeff47f9796cb24164489fae94cfbc4b564c0f0329d9774add8ce54"} Nov 01 21:43:55.506106 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:55.505923 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" podStartSLOduration=6.074598581 podStartE2EDuration="21.505910462s" totalImagesPullingTime="15.431311881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:55.504015221 +0000 UTC m=+22.830034265" watchObservedRunningTime="2026-11-01 21:43:55.505910462 +0000 UTC m=+22.831929496" Nov 01 21:43:56.468615 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:56.468590 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:56.469152 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:56.469059 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:56.469223 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:56.469160 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:56.485125 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:56.485104 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:56.485242 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:56.485215 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:43:57.115182 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.115153 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4w9sx"] Nov 01 21:43:57.115367 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.115289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:57.115433 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:57.115397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:57.118093 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.118064 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs8p9"] Nov 01 21:43:57.118220 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.118187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:57.118321 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:57.118297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:57.118653 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.118630 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qghnn"] Nov 01 21:43:57.118781 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.118726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:57.119466 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:57.119424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:43:57.470522 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.470495 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:57.695461 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.695432 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:57.695615 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.695530 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:57.696123 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:57.696104 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:43:58.475539 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:58.475514 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:43:59.194116 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:59.194085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:43:59.194116 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:59.194112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:43:59.194316 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:43:59.194085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:43:59.194316 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:59.194193 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:43:59.194316 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:59.194287 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:43:59.194409 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:43:59.194355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:44:01.194465 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.194429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:01.194943 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.194442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:01.194943 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:01.194549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:44:01.194943 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:01.194639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:44:01.194943 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.194688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:01.194943 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:01.194741 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:44:01.502560 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.502533 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:01.502755 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.502742 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 21:44:01.513484 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.513447 2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" podUID="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 21:44:01.521969 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:01.521943 2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" podUID="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 21:44:03.194853 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:03.194705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:03.195256 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:03.194908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4w9sx" podUID="6eaffc4a-a597-4820-8a11-31e92606f9f2" Nov 01 21:44:03.195256 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:03.194814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:03.195256 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:03.194975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qghnn" podUID="a0158144-5a21-4098-869a-dd1598f3aad8" Nov 01 21:44:03.195256 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:03.194789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:03.195256 ip-10-0-136-47 kubenswrapper[2576]: E1101 21:44:03.195062 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qs8p9" podUID="c83a4e94-9b7d-435a-bd02-b043476da87a" Nov 01 21:44:03.429647 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:03.429592 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeReady" Nov 01 21:44:03.429747 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:03.429683 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 21:44:04.803793 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:04.803770 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mc686_eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0/dns-node-resolver/0.log" Nov 01 21:44:05.194687 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.194599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:05.194687 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.194617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:05.194896 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.194813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:05.200322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200232 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 21:44:05.200322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200244 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 21:44:05.200322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200265 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 21:44:05.200322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200272 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\"" Nov 01 21:44:05.200322 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200323 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 21:44:05.200842 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200333 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\"" Nov 01 21:44:05.200842 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.200391 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\"" Nov 01 21:44:05.573888 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:05.573861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbcqn_4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5/node-ca/0.log" Nov 01 21:44:06.338771 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.338735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wgdj_5ff79f3e-c818-4103-b626-3bed7d7b46b4/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:06.840279 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.840244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:06.840433 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.840293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:06.844325 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.844295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:06.844432 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.844403 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:06.941446 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.941414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:06.943766 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:06.943750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:07.008040 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.007999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:07.017117 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.017094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:07.020253 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.020235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:07.124383 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.124356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7wgdj_5ff79f3e-c818-4103-b626-3bed7d7b46b4/konnectivity-agent/0.log" limit=1024 Nov 01 21:44:07.303106 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.303077 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qghnn"] Nov 01 21:44:07.306456 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:44:07.306424 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0158144_5a21_4098_869a_dd1598f3aad8.slice/crio-189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f WatchSource:0}: Error finding container 189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f: Status 404 returned error can't find the container with id 189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f Nov 01 21:44:07.307613 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.307574 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4w9sx"] Nov 01 21:44:07.308351 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.308324 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qs8p9"] Nov 01 21:44:07.311133 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:44:07.311110 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83a4e94_9b7d_435a_bd02_b043476da87a.slice/crio-9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a WatchSource:0}: Error finding container 9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a: Status 404 returned error can't find the container with id 9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a Nov 01 21:44:07.311674 ip-10-0-136-47 kubenswrapper[2576]: W1101 21:44:07.311657 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaffc4a_a597_4820_8a11_31e92606f9f2.slice/crio-d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022 WatchSource:0}: Error finding container d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022: Status 404 returned error can't find the container with id d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022 Nov 01 21:44:07.496626 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.496593 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qghnn" event={"ID":"a0158144-5a21-4098-869a-dd1598f3aad8","Type":"ContainerStarted","Data":"189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f"} Nov 01 21:44:07.497899 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.497874 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs8p9" event={"ID":"c83a4e94-9b7d-435a-bd02-b043476da87a","Type":"ContainerStarted","Data":"9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a"} Nov 01 21:44:07.499105 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:07.499084 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4w9sx" event={"ID":"6eaffc4a-a597-4820-8a11-31e92606f9f2","Type":"ContainerStarted","Data":"d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022"} Nov 01 21:44:09.506308 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:09.506271 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs8p9" event={"ID":"c83a4e94-9b7d-435a-bd02-b043476da87a","Type":"ContainerStarted","Data":"5593956c1215ecae1a67a8429f1f9f80ce496b962e91c95d9b578cf88d7db16f"} Nov 01 21:44:09.506745 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:09.506325 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qs8p9" event={"ID":"c83a4e94-9b7d-435a-bd02-b043476da87a","Type":"ContainerStarted","Data":"4a6e313d7f5ce020f50066ed27615be74cd5a6190fcfc7e86647777d19242687"} Nov 01 21:44:11.513496 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:11.513457 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qghnn" event={"ID":"a0158144-5a21-4098-869a-dd1598f3aad8","Type":"ContainerStarted","Data":"e089319673c549a7a8c2572e6ed96a8c7f58fe0e1566138e014248e8cf7a33ac"} Nov 01 21:44:11.514004 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:11.513739 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:11.532998 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:11.532943 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qs8p9" podStartSLOduration=35.807021324 podStartE2EDuration="37.532927434s" totalImagesPullingTime="1.72590611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:09.526579821 +0000 UTC m=+36.852598856" watchObservedRunningTime="2026-11-01 21:44:11.532927434 +0000 UTC m=+38.858946469" Nov 01 21:44:11.533392 ip-10-0-136-47 kubenswrapper[2576]: I1101 21:44:11.533365 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qghnn" podStartSLOduration=33.597149877 podStartE2EDuration="37.533360215s" totalImagesPullingTime="3.936210338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:11.53211431 +0000 UTC m=+38.858133348" watchObservedRunningTime="2026-11-01 21:44:11.533360215 +0000 UTC m=+38.859379247" Nov 01 21:44:12.965839 ip-10-0-136-47 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 21:44:12.975409 ip-10-0-136-47 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 21:44:12.975570 ip-10-0-136-47 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 21:44:12.975617 ip-10-0-136-47 systemd[1]: kubelet.service: Consumed 2.086s CPU time. Nov 01 21:44:12.996693 ip-10-0-136-47 systemd[1]: Starting Kubernetes Kubelet... Nov 01 21:44:13.080312 ip-10-0-136-47 kubenswrapper[4395]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:13.080312 ip-10-0-136-47 kubenswrapper[4395]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 21:44:13.080312 ip-10-0-136-47 kubenswrapper[4395]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:13.080312 ip-10-0-136-47 kubenswrapper[4395]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 21:44:13.082573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082499 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.082573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082569 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.082573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082575 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.082573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082580 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082584 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082588 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082592 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082595 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082599 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082603 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082606 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082610 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082614 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082618 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082622 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082626 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082630 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082634 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082637 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082641 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082644 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082648 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082658 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.082847 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082662 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082666 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082680 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082686 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082690 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082694 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082698 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082702 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082706 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082709 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082714 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082718 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082722 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082727 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082732 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082739 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082745 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082750 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082755 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.083772 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082760 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082765 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082769 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082774 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082778 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082782 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082786 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082790 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082794 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082799 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082803 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082807 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082811 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082816 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082820 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082825 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082829 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082834 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082838 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082842 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.084640 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082846 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082850 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082854 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082860 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082864 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082868 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082872 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082877 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082882 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082886 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082891 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082895 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082899 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082903 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082907 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082911 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082915 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082919 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082923 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.085520 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082927 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082932 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082936 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082940 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082945 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082949 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082953 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082958 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082962 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082966 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082971 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082975 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082980 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082984 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082988 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082992 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.082996 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083000 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083004 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083008 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.086431 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083013 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083018 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083039 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083044 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.083051 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084260 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084269 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084274 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084278 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084283 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084287 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084291 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084295 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084299 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084303 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084307 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084312 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084316 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084320 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.087288 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084324 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084329 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084334 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084338 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084342 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084347 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084351 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084355 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084359 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084363 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084367 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084371 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084376 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084381 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084386 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084390 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084394 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084398 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084402 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.088087 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084406 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084410 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084414 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084418 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084422 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084426 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084430 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084434 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084438 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084442 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084446 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084450 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084454 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084458 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084463 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084468 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084472 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084479 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084485 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.088581 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084491 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084497 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084502 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084507 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084511 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084516 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084520 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084524 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084529 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084533 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084537 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084541 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084545 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084550 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084554 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084558 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084562 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084566 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084570 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.089084 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084575 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084579 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084583 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084587 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084591 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084595 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084599 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084604 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084608 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084612 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084618 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084623 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084627 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084631 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084635 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084639 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084643 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084646 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084651 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084655 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.089552 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084659 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084663 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084667 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084672 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084676 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084680 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084684 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084689 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084693 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084697 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084701 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084706 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084709 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084713 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.084717 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085531 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085540 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085544 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085549 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085553 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.090069 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085558 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085562 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085568 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085572 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085576 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085580 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085585 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085588 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085592 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085597 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085601 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085605 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085612 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085617 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085623 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085628 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085632 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085636 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085641 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.090553 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085645 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085649 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085653 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085658 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085663 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085667 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085671 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085675 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085679 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085683 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085687 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085691 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085696 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085700 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085704 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085709 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085713 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085718 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085722 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.091085 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085726 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085730 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085734 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085738 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085742 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085747 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085751 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085757 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085762 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085767 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085771 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085775 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085780 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085784 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085788 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085792 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085796 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085802 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085806 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085810 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.091571 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085814 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085818 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085822 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085826 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085830 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085834 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085838 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085842 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085847 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085851 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085856 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085860 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085864 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085869 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085873 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085877 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085881 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085885 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085889 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085893 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.092073 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085897 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085901 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085905 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085908 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085912 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085917 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085921 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085925 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085929 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085933 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085937 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085943 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085947 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085951 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085955 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085959 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085963 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085967 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085971 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085975 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.092573 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085983 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085987 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.085991 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086143 4395 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086155 4395 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086164 4395 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086171 4395 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086177 4395 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086182 4395 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086188 4395 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086194 4395 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086199 4395 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086203 4395 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086209 4395 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086214 4395 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086219 4395 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086224 4395 flags.go:64] FLAG: --cgroup-root="" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086228 4395 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086233 4395 flags.go:64] FLAG: --client-ca-file="" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086238 4395 flags.go:64] FLAG: --cloud-provider="external" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086242 4395 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086247 4395 flags.go:64] FLAG: --cluster-domain="" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086252 4395 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 21:44:13.093079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086256 4395 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086261 4395 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086268 4395 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086274 4395 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086278 4395 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086283 4395 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086288 4395 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086292 4395 flags.go:64] FLAG: --contention-profiling="false" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086297 4395 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086304 4395 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086310 4395 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086314 4395 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086321 4395 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086326 4395 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086331 4395 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086335 4395 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086340 4395 flags.go:64] FLAG: --enable-server="true" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086345 4395 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086351 4395 flags.go:64] FLAG: --event-burst="100" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086356 4395 flags.go:64] FLAG: --event-qps="50" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086361 4395 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086365 4395 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086370 4395 flags.go:64] FLAG: --eviction-hard="" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086376 4395 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 21:44:13.093632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086381 4395 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086385 4395 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086390 4395 flags.go:64] FLAG: --eviction-soft="" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086395 4395 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086402 4395 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086406 4395 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086411 4395 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086416 4395 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086422 4395 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086426 4395 flags.go:64] FLAG: --feature-gates="" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086432 4395 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086437 4395 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086442 4395 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086446 4395 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086451 4395 flags.go:64] FLAG: --healthz-port="10248" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086455 4395 flags.go:64] FLAG: --help="false" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086460 4395 flags.go:64] FLAG: --hostname-override="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086468 4395 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086473 4395 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086477 4395 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086483 4395 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086488 4395 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086497 4395 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 21:44:13.094241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086502 4395 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086506 4395 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086511 4395 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086515 4395 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086520 4395 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086525 4395 flags.go:64] FLAG: --kube-reserved="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086529 4395 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086534 4395 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086538 4395 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086542 4395 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086547 4395 flags.go:64] FLAG: --lock-file="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086552 4395 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086557 4395 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086563 4395 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086572 4395 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086576 4395 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086581 4395 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086585 4395 flags.go:64] FLAG: --logging-format="text" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086590 4395 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086595 4395 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086600 4395 flags.go:64] FLAG: --manifest-url="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086604 4395 flags.go:64] FLAG: --manifest-url-header="" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086611 4395 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086616 4395 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086622 4395 flags.go:64] FLAG: --max-pods="110" Nov 01 21:44:13.094809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086627 4395 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086633 4395 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086638 4395 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086643 4395 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086648 4395 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086652 4395 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086657 4395 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086668 4395 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086680 4395 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086685 4395 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086690 4395 flags.go:64] FLAG: --pod-cidr="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086695 4395 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086699 4395 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086704 4395 flags.go:64] FLAG: --pods-per-core="0" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086708 4395 flags.go:64] FLAG: --port="10250" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086713 4395 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086717 4395 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0cfe303a03f2d5efb" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086722 4395 flags.go:64] FLAG: --qos-reserved="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086727 4395 flags.go:64] FLAG: --read-only-port="10255" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086731 4395 flags.go:64] FLAG: --register-node="true" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086739 4395 flags.go:64] FLAG: --register-with-taints="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086745 4395 flags.go:64] FLAG: --registry-burst="10" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086750 4395 flags.go:64] FLAG: --registry-qps="5" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086754 4395 flags.go:64] FLAG: --reserved-cpus="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086758 4395 flags.go:64] FLAG: --reserved-memory="" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086765 4395 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 21:44:13.095456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086770 4395 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086774 4395 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086779 4395 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086784 4395 flags.go:64] FLAG: --runonce="false" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086789 4395 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086793 4395 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086798 4395 flags.go:64] FLAG: --seccomp-default="false" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086804 4395 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086809 4395 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086814 4395 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086818 4395 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086823 4395 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086828 4395 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086832 4395 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086838 4395 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086842 4395 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086847 4395 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086852 4395 flags.go:64] FLAG: --system-cgroups="" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086856 4395 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086865 4395 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086869 4395 flags.go:64] FLAG: --tls-cert-file="" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086874 4395 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086880 4395 flags.go:64] FLAG: --tls-min-version="" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086884 4395 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086889 4395 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 21:44:13.096167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086893 4395 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086899 4395 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086904 4395 flags.go:64] FLAG: --v="2" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086910 4395 flags.go:64] FLAG: --version="false" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086916 4395 flags.go:64] FLAG: --vmodule="" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086922 4395 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.086927 4395 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.087989 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088002 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088008 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088013 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088075 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088099 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088103 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088128 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088146 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088277 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088286 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088289 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088292 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088295 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.096872 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088298 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088300 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088303 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088306 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088309 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088311 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088314 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088317 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088319 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088322 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088325 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088328 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088332 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088337 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088341 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088344 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088348 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088351 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088354 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.097517 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088357 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088360 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088363 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088366 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088368 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088371 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088374 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088377 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088379 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088382 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088384 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088387 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088390 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088392 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088395 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088397 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088400 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088402 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088405 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088408 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.098079 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088410 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088413 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088416 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088421 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088423 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088426 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088429 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088431 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088434 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088436 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088439 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088441 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088444 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088446 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088448 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088451 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088454 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088457 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088459 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088462 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.098611 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088465 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088469 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088472 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088475 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088477 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088480 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088482 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088485 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088487 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088489 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088493 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088496 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088499 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088501 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088504 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088508 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088511 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088514 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088516 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088519 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.099277 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088522 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088524 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088527 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088529 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088532 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088535 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088537 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088540 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088542 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088545 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088547 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088550 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088552 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.088591 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088639 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.099865 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088644 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088647 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088650 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088653 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088657 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088660 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088662 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088665 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088677 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088680 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088682 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088685 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088689 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088691 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088694 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088696 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088699 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088702 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088704 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.100377 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088707 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088709 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088712 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088714 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088717 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088719 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088722 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088724 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088727 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088729 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088732 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088734 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088736 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088739 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088741 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088744 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088746 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088749 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088751 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.100916 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088754 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088758 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088761 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088770 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088773 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088775 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088779 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088782 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088784 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088787 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088789 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088791 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088794 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088797 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088799 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088801 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088804 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088806 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088809 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088811 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.101404 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088814 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088816 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088818 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088821 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088823 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088827 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088830 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088834 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088836 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088839 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088841 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088844 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088847 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088849 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088853 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088855 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088858 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088862 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088866 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.101890 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088869 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088872 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088875 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088877 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088880 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088882 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088885 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088887 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088890 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088892 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088895 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088897 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088900 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088902 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088904 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088907 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088909 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088912 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088914 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088916 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.102541 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088919 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088922 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088924 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088926 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088929 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088931 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088934 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.088939 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.092701 4395 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.092716 4395 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.092733 4395 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.092738 4395 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092791 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092797 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092800 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092804 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092807 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092810 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092813 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092816 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.103052 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092818 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092821 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092823 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092826 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092829 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092831 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092834 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092836 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092839 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092841 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092844 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092847 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092849 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092852 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092856 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092860 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092863 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092865 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092868 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.103540 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092871 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092873 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092877 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092879 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092882 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092884 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092887 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092889 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092892 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092894 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092897 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092900 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092902 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092905 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092908 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092910 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092913 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092915 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092917 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.104008 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092920 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092923 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092925 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092928 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092930 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092933 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092936 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092938 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092941 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092944 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092947 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092949 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092952 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092954 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092957 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092959 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092962 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092965 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092967 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.104587 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092970 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092973 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092975 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092977 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092980 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092982 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092985 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092987 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092990 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092992 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092995 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.092997 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093000 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093002 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093004 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093007 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093009 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093012 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093014 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093017 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.105082 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093020 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093022 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093043 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093046 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093049 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093051 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093054 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093057 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093059 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093062 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093065 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093068 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093070 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093073 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093076 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093078 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093080 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093084 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093089 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.105557 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093091 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093094 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.093121 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093272 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093278 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093281 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093283 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093286 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093289 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093292 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093295 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093298 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093302 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093305 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093308 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.106193 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093311 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093313 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093316 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093318 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093321 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093323 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093326 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093328 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093331 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093333 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093336 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093338 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093341 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093343 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093346 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093349 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093351 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093353 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093356 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.106804 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093358 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093360 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093363 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093365 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093367 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093370 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093372 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093374 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093377 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093379 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093382 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093385 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093387 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093389 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093392 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093394 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093397 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093399 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093402 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093405 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.107415 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093407 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093409 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093412 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093420 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093423 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093425 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093428 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093430 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093432 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093435 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093437 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093440 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093442 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093444 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093447 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093449 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093453 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093456 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093459 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.108102 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093461 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093464 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093466 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093469 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093472 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093474 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093477 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093479 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093482 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093484 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093486 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093489 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093491 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093493 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093496 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093498 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093500 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093503 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093506 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093508 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.108595 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093511 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093513 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093515 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093518 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093520 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093522 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093525 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093529 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093532 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093534 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093537 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093539 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093542 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093544 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093546 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093549 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.109101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.093572 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093602 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093605 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093608 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093611 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093614 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093616 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093619 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093621 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093624 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093626 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093628 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093631 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093633 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093636 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093639 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093642 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093644 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093647 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093650 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 21:44:13.109644 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093652 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093655 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093657 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093660 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093663 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093667 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093670 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093672 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093675 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093678 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093680 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093683 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093686 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093688 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093691 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093693 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093695 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093698 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093700 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 21:44:13.110477 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093702 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093705 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093707 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093710 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093712 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093714 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093717 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093719 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093722 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093725 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093727 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093729 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093732 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093734 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093737 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093739 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093743 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093746 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093748 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093751 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 21:44:13.111342 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093753 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093756 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093758 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093761 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093763 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093766 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093768 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093770 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093773 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093775 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093778 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093780 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093783 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093788 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093790 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093792 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093795 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093798 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093800 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 21:44:13.111908 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093803 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093805 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093808 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093810 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093813 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093815 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093818 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093820 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093822 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093825 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093827 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093830 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093832 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093834 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093837 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093839 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093841 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093844 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093846 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 21:44:13.112423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093849 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093851 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093853 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093856 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093858 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093861 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093864 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093869 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093872 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:13.093874 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.093925 4395 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.094876 4395 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.094935 4395 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.095133 4395 server.go:1014] "Starting client certificate rotation" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.095271 4395 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.095322 4395 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:33 +0000 UTC" deadline="2028-04-11 07:44:16.6785784 +0000 UTC" Nov 01 21:44:13.112873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.095348 4395 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12634h0m3.583232171s" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.095599 4395 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.096700 4395 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.098188 4395 log.go:25] "Validated CRI v1 runtime API" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.099639 4395 log.go:25] "Validated CRI v1 image API" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.100385 4395 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 21:44:13.113293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.101781 4395 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d76ff796-256e-467c-9aa4-af9533a85cd9:/dev/nvme0n1p4 dbb5e605-2cc4-4937-8e3a-42faa5b4001b:/dev/nvme0n1p3] Nov 01 21:44:13.113455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.101796 4395 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fe1ba52-ea23-43ce-9286-975b58f3998d/volumes/kubernetes.io~projected/kube-api-access-cpwrg:{mountpoint:/var/lib/kubelet/pods/0fe1ba52-ea23-43ce-9286-975b58f3998d/volumes/kubernetes.io~projected/kube-api-access-cpwrg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b809a87-86be-4c57-a0e5-e491568d34c1/volumes/kubernetes.io~projected/kube-api-access-rqfqs:{mountpoint:/var/lib/kubelet/pods/3b809a87-86be-4c57-a0e5-e491568d34c1/volumes/kubernetes.io~projected/kube-api-access-rqfqs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5/volumes/kubernetes.io~projected/kube-api-access-dpxpg:{mountpoint:/var/lib/kubelet/pods/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5/volumes/kubernetes.io~projected/kube-api-access-dpxpg major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~projected/kube-api-access-8sww5:{mountpoint:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~projected/kube-api-access-8sww5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ff79f3e-c818-4103-b626-3bed7d7b46b4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5ff79f3e-c818-4103-b626-3bed7d7b46b4/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eaffc4a-a597-4820-8a11-31e92606f9f2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6eaffc4a-a597-4820-8a11-31e92606f9f2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0158144-5a21-4098-869a-dd1598f3aad8/volumes/kubernetes.io~projected/kube-api-access-v6d62:{mountpoint:/var/lib/kubelet/pods/a0158144-5a21-4098-869a-dd1598f3aad8/volumes/kubernetes.io~projected/kube-api-access-v6d62 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1177d05-06de-4967-b190-b00451f8f853/volumes/kubernetes.io~projected/kube-api-access-9t8vx:{mountpoint:/var/lib/kubelet/pods/b1177d05-06de-4967-b190-b00451f8f853/volumes/kubernetes.io~projected/kube-api-access-9t8vx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~projected/kube-api-access-66wr8:{mountpoint:/var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~projected/kube-api-access-66wr8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~projected/kube-api-access-7z9fn:{mountpoint:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~projected/kube-api-access-7z9fn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0/volumes/kubernetes.io~projected/kube-api-access-4trdc:{mountpoint:/var/lib/kubelet/pods/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0/volumes/kubernetes.io~projected/kube-api-access-4trdc major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed118150-74ab-4355-8f26-22d188e0140b/volumes/kubernetes.io~projected/kube-api-access-5h25l:{mountpoint:/var/lib/kubelet/pods/ed118150-74ab-4355-8f26-22d188e0140b/volumes/kubernetes.io~projected/kube-api-access-5h25l major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d7ab51ab95f4017eae7c829eefb0548912e1298f8ea3092cf6058b35be8a5cd5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0dbc27802494ca6e6463d473ef360e67c4f7e39fd98f50b2c0f3f4ae281b2aff/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3365b83fb21e08c6c3247dc46078a8d5f85817a6562a95efefb047904533b39a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a8d96caca71ed0904cab8eb01b12a4e0e4373afd5d439179fff39faf80522f98/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9bb2c18dc3ffc080544c687d41330231885269439aeb847f605f04a8f78e4463/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/233306bcea0be27de0fd3847f85f24927ec28db2a8201b4cdbf7cc718a452e99/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a69874c0210b673bb0cf913a775a511b77f3faa794324177a4de654249957858/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8c17810cbaa198464a5d630558764343b49da6a9fbc0ca2426f6ed2e7288ba87/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2146d89a51d7b77ce72768c1efbb1adc46aeb775f9dd5f7e6471c74afa5ad2af/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6da19b685018e09f8b39009777c224b463f37582b30a95a97ee3b3156bb84771/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/3de3de8393343aff76b5c52f77029139c2fe3f19d565f355f068528308a80c44/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/971005a069bb15b4d78d5d5d5dd45c45a65b62360f8814937f0a98d214d32be2/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/53b022a73586903cd01d478da8edd902a93e298ebab57573ebd06c252f51999e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/63dbff8414621e7c95ad2abce3d705c7de22d10892add23ee17cd7bf51cef981/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/16a61b08ec0cfb3c8de1e373d3fcb9bb7647c7460fd6f4bf4c855b18f10faa81/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8b97f401760b35e48e7e4d3cafd619ea5361050428e309615785b75ab10064de/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5d178a3cce9bfbb9944f53384c397dbb3ddf7e06845e30a9b858d41714edd89f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/edfc64354ba698892a15a47e979ead1740830f27d1ca97cce3ced7c8280b075f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1a9ab7944eba2dbd468a24294c09b7143a39421505cfd41e18fc670d21e49fd8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2932919ca6ceca10135436836ced7ff373554c993aff81db971dfcfb2c4185e1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a26096b383420237f829b221f4692879da1083525171096b6c8d6ff65e97d667/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/ab6685686b375be700e6b408eb6cce0c91bd70d1b08811596cf0f6d2efdca339/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/5f07494cd880da0b24ba55ac4db7af6cc17af13714bf8770cbc430bf16d6498f/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/b995524fd28733e6f9aefa601f67c4036ad15008c6c6059e6ca9d12b96cab675/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/48a2b505c9b5fba4dd5e254ef1b440e6014938c2200c63275ebdf1992dc4f0a7/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/125fc2173d3bac27e28d9d5a179e9801cbad9bd43d68a236489e5544ac0ef07e/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/c2b3511d00cd9fe31288d72e42478e3a6d651477a9836515ce3acb52dda4da38/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/d873c16e220ed3b94cbc0852b67b7e1cbdd27e5d1515f4eb38775618e9df3cdd/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/604a70c3ad916511c2051893a1f5e23323121a599f5df15ef15fa2a8c5e908c6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/32fe4131f338bb496f91cb7999ebf577ebcee5395242bbdf54b6dcaa644e9ff1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1d8da71202e9d4515d311f5136c386d77760e5d7a1e7450fbc255c402ce4c6c1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4f6b321a83c5601cdf0ad60eb9f5ae5984d6381a0fe9af7a63f4fdb5b2b54701/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bb7a15add37fdcbec11c8896e55344f89bc26ac866efe716fee6c863ebc65350/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5c48c62d8fde426a3698f6873cce3c05669c8664363238df8cd300cd3587a86d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c4ad2808851c3e8c8717e80ee343769c8ac03e57bec7e269a1c6283b3a757a5c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/635f76861cb31babfbda07b47f0cf4722150636202d50ba36fb4ec74e8aed079/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/bbb2492c4848bdc31c1ea3a274c8c8074dea3052e27c300511ace7d26b2e5704/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/12cc198f667ed52d3b79ff186673b9bf2f2d912f9cb17b6d84c7607a0304c4b2/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 01 21:44:13.113455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.108973 4395 manager.go:217] Machine: {Timestamp:2026-11-01 21:44:13.108534161 +0000 UTC m=+0.082436690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122100 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2778a09e19b7ac909172f8ad5b4549 SystemUUID:ec2778a0-9e19-b7ac-9091-72f8ad5b4549 BootID:64564fa0-f11e-4a12-8339-2bc6743cad34 Filesystems:[{Device:/run/containers/storage/overlay-containers/ae7d02791d3eab9e53864869618f2bf46985c2fe1a3885ff932a2b66d74bb0e2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66403a367493b8066b5b5455c3bce9043787ccccf6bc45f41805da2900ef4314/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0/volumes/kubernetes.io~projected/kube-api-access-4trdc DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~projected/kube-api-access-66wr8 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~projected/kube-api-access-7z9fn DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~projected/kube-api-access-8sww5 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0fe1ba52-ea23-43ce-9286-975b58f3998d/volumes/kubernetes.io~projected/kube-api-access-cpwrg DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c1f374bba605bebc9fee8feed3b9d29c97873094cd39db9ee26c9ed02329b2a/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/708910a5d0c61167a1d0f852a44a9e111dddb2fdf1f7177b22ad9cb22b2f268e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b809a87-86be-4c57-a0e5-e491568d34c1/volumes/kubernetes.io~projected/kube-api-access-rqfqs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0eb95ae916a22bf4a9a3faeef0ac110726b93c5a90c259ea25801f315f01813/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8c9f32bf9734878af9cc74573cc395de9d1bf29cfb85da8ece6b141fcbc611a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8900b3f0b27372a371a80a35a01c763764a4f70b40ffecafa39d1fadd8baa462/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0158144-5a21-4098-869a-dd1598f3aad8/volumes/kubernetes.io~projected/kube-api-access-v6d62 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6eaffc4a-a597-4820-8a11-31e92606f9f2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0926b2ab1459a0b1988b39eac727eec26871faee9daae6e21e09b4af4a16ef87/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9f841f8bf29384f43bd68cba09cc33b133803606ea4c09fbd758dc13f662022/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/189dd0d604a4879ce92f3f919d402d4f54fc7a5f66584eeefcf68098e87ab98f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbfa103013dfdd03fb9d8f501766fa9b695300f6c459f44803a72f1a28a1955c/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ff79f3e-c818-4103-b626-3bed7d7b46b4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed118150-74ab-4355-8f26-22d188e0140b/volumes/kubernetes.io~projected/kube-api-access-5h25l DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1177d05-06de-4967-b190-b00451f8f853/volumes/kubernetes.io~projected/kube-api-access-9t8vx DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c83a4e94-9b7d-435a-bd02-b043476da87a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0464581-54a9-4673-a69c-3b560089a3c4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5/volumes/kubernetes.io~projected/kube-api-access-dpxpg DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:189dd0d604a4879 MacAddress:fa:68:3c:5d:30:cd Speed:10000 Mtu:8901} {Name:9c1f374bba605be MacAddress:d6:e5:4a:c5:5b:70 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:6f:52:75 Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:a5:08:ab:a6:96 Speed:0 Mtu:8901} {Name:d9f841f8bf29384 MacAddress:b2:e1:59:2f:1c:8f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:6f:52:75 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:75:7a:c5:a4:b6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:b3:99:00:19:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 21:44:13.113455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109204 4395 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 21:44:13.113455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109245 4395 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 21:44:13.113455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109572 4395 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109597 4395 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-47.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109776 4395 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109788 4395 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109806 4395 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109833 4395 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.109928 4395 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110154 4395 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110299 4395 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110342 4395 kubelet.go:493] "Attempting to sync node with API server" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110361 4395 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110374 4395 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110386 4395 kubelet.go:405] "Adding apiserver pod source" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.110397 4395 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.111169 4395 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.111181 4395 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.111647 4395 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113504 4395 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113513 4395 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 21:44:13.113753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113602 4395 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113771 4395 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113797 4395 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113878 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113890 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113896 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113901 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113906 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113912 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113917 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113922 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113928 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113933 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113946 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113955 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113977 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 21:44:13.114310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.113982 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 21:44:13.117173 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.117158 4395 server.go:1263] "Started kubelet" Nov 01 21:44:13.117264 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.117229 4395 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 21:44:13.117501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.117371 4395 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 21:44:13.117501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.117445 4395 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 21:44:13.117674 ip-10-0-136-47 systemd[1]: Started Kubernetes Kubelet. Nov 01 21:44:13.117805 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.117727 4395 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 21:44:13.118445 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.118431 4395 server.go:317] "Adding debug handlers to kubelet server" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125160 4395 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125609 4395 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125623 4395 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.125739 4395 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-47.us-east-2.compute.internal\" not found" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125754 4395 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125612 4395 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125891 4395 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:35 +0000 UTC" deadline="2028-07-26 22:28:16.93487658 +0000 UTC" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.125923 4395 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15192h44m3.808957518s" Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.126678 4395 factory.go:55] Registering systemd factory Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.126703 4395 factory.go:223] Registration of the systemd container factory successfully Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.126938 4395 factory.go:153] Registering CRI-O factory Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.126949 4395 factory.go:223] Registration of the crio container factory successfully Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127007 4395 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127045 4395 factory.go:103] Registering Raw factory Nov 01 21:44:13.127337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127075 4395 manager.go:1199] Started watching for new ooms in manager Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127418 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83a4e94-9b7d-435a-bd02-b043476da87a" volumeName="kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127461 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0464581-54a9-4673-a69c-3b560089a3c4" volumeName="kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127465 4395 manager.go:319] Starting recovery of all containers Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127478 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5" volumeName="kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127536 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5" volumeName="kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127587 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" volumeName="kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127631 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" volumeName="kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127653 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1177d05-06de-4967-b190-b00451f8f853" volumeName="kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127674 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0464581-54a9-4673-a69c-3b560089a3c4" volumeName="kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127694 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed118150-74ab-4355-8f26-22d188e0140b" volumeName="kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127707 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fe1ba52-ea23-43ce-9286-975b58f3998d" volumeName="kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127725 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" volumeName="kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127737 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ff79f3e-c818-4103-b626-3bed7d7b46b4" volumeName="kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127749 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83a4e94-9b7d-435a-bd02-b043476da87a" volumeName="kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127764 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0" volumeName="kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127823 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed118150-74ab-4355-8f26-22d188e0140b" volumeName="kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127852 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed118150-74ab-4355-8f26-22d188e0140b" volumeName="kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127864 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b809a87-86be-4c57-a0e5-e491568d34c1" volumeName="kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 21:44:13.128221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.127882 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b809a87-86be-4c57-a0e5-e491568d34c1" volumeName="kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128421 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" volumeName="kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128447 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ff79f3e-c818-4103-b626-3bed7d7b46b4" volumeName="kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128458 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0158144-5a21-4098-869a-dd1598f3aad8" volumeName="kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128466 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1177d05-06de-4967-b190-b00451f8f853" volumeName="kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128473 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0464581-54a9-4673-a69c-3b560089a3c4" volumeName="kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128485 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0" volumeName="kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128497 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b809a87-86be-4c57-a0e5-e491568d34c1" volumeName="kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128510 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b809a87-86be-4c57-a0e5-e491568d34c1" volumeName="kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128522 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac" volumeName="kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128535 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eaffc4a-a597-4820-8a11-31e92606f9f2" volumeName="kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret" seLinuxMountContext="" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128565 4395 reconstruct.go:97] "Volume reconstruction finished" Nov 01 21:44:13.128902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.128572 4395 reconciler.go:29] "Reconciler: start to sync state" Nov 01 21:44:13.129534 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.129421 4395 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:13.131220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.131200 4395 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:13.131521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.131506 4395 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:13.132954 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.132938 4395 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 21:44:13.142904 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.142867 4395 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 21:44:13.156814 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.156790 4395 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 21:44:13.156814 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.156815 4395 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 21:44:13.156989 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.156840 4395 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 21:44:13.156989 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.156882 4395 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 21:44:13.160580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.160561 4395 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166895 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="d4464094dfd9cce4407b055296e69df5502d1255121722e12b890b1c910960f2" exitCode=0 Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166923 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="a4411d0b19fece45da98305dae575a65e95da0d07b9e271c57589ff12ee1f716" exitCode=0 Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166933 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="818c1193dadbc2dea8ecf16cde9747877dd6d25aefb30eced3d7f965d9f83385" exitCode=0 Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166944 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="972bdc355742a902915e25ea112ee2faf5a8c72f453d89174fc5ee4d5b2cf5ee" exitCode=0 Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166954 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="e261e5777668766d9eca6bd99a089ffc6d842bb6ea576de41a33e950ba422364" exitCode=0 Nov 01 21:44:13.168434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.166964 4395 generic.go:356] "Generic (PLEG): container finished" podID="3b809a87-86be-4c57-a0e5-e491568d34c1" containerID="3c2452ff61ff172b0d82faa54a97e8754aecb89483f0b616c8ea46e7fa7cc9cb" exitCode=0 Nov 01 21:44:13.172063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.172043 4395 generic.go:356] "Generic (PLEG): container finished" podID="f04a04cb275f35547de27e44efd5af4f" containerID="d7e7f3bd94257ebe4930530d6258cbde237424c345ee9d8cf79568ffb9c8f928" exitCode=0 Nov 01 21:44:13.192451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.192431 4395 manager.go:324] Recovery completed Nov 01 21:44:13.205657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205638 4395 cpu_manager.go:226] "Starting" policy="none" Nov 01 21:44:13.205657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205652 4395 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 21:44:13.205796 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205667 4395 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 21:44:13.205796 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205779 4395 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205794 4395 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205814 4395 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205819 4395 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205824 4395 policy_none.go:50] "Start" Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205832 4395 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 21:44:13.205859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205841 4395 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 21:44:13.206019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205973 4395 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 21:44:13.206019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205980 4395 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 21:44:13.206019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.205985 4395 policy_none.go:44] "Start" Nov 01 21:44:13.209936 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.209909 4395 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 21:44:13.212447 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.212435 4395 manager.go:341] "Starting Device Plugin manager" Nov 01 21:44:13.213161 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.212471 4395 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 21:44:13.213161 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213162 4395 server.go:87] "Starting device plugin registration server" Nov 01 21:44:13.213413 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213403 4395 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 21:44:13.213464 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213414 4395 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 21:44:13.213549 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213530 4395 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 21:44:13.213655 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213622 4395 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 21:44:13.213821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213663 4395 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 21:44:13.213821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213671 4395 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 21:44:13.213821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.213713 4395 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:44:13.213652748Z","UUID":"975dd97a-ee6d-47aa-a292-b021b9987f23","Handler":null,"Name":"","Endpoint":""} Nov 01 21:44:13.214837 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.214821 4395 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 21:44:13.215285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.215265 4395 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 21:44:13.215383 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.215304 4395 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 21:44:13.257509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.257470 4395 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal"] Nov 01 21:44:13.257971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.257935 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"91729725e20792f4990b48a3f50d1312","Type":"ContainerStarted","Data":"acaa439727a289502928e2c7305c4169b20d51f6bde5f586fe4c51a8442d4afe"} Nov 01 21:44:13.258044 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.257986 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"91729725e20792f4990b48a3f50d1312","Type":"ContainerStarted","Data":"6b1712ee743bea506740cc00087992e3a36085144f8cca38327b2d37f8df2499"} Nov 01 21:44:13.258044 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.258010 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerStarted","Data":"070b0c1ba29bbefda936de123daf34f3246171324456ac0af4654d2f1f3f9ba9"} Nov 01 21:44:13.258114 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.258056 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerDied","Data":"d7e7f3bd94257ebe4930530d6258cbde237424c345ee9d8cf79568ffb9c8f928"} Nov 01 21:44:13.258114 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.258081 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" event={"ID":"f04a04cb275f35547de27e44efd5af4f","Type":"ContainerStarted","Data":"c0afacb08834e890aaad8f5a7d7217bb2fea237546e5c3e19b82f1dac537256b"} Nov 01 21:44:13.258114 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.258086 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.258238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.258223 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.270926 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.270910 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:13.270977 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.270952 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.271505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.271492 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:13.271546 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:13.271526 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.313634 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.313617 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 21:44:13.314835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.314821 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 21:44:13.314902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.314848 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 21:44:13.314902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.314859 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-47.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 21:44:13.314902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.314896 4395 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.333088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.333069 4395 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.333144 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.333104 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.333144 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.333127 4395 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.333209 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.333133 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.333209 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.333170 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433529 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433546 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433573 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433610 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f04a04cb275f35547de27e44efd5af4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"f04a04cb275f35547de27e44efd5af4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:13.433695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:13.433625 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91729725e20792f4990b48a3f50d1312-config\") pod \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" (UID: \"91729725e20792f4990b48a3f50d1312\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:14.111078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.111045 4395 apiserver.go:52] "Watching apiserver" Nov 01 21:44:14.118872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.118849 4395 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 21:44:14.119208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.119191 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7wgdj","openshift-cluster-node-tuning-operator/tuned-nf8vf","openshift-image-registry/node-ca-gbcqn","openshift-network-operator/iptables-alerter-c2ntz","openshift-multus/network-metrics-daemon-qs8p9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg","openshift-dns/node-resolver-mc686","openshift-network-diagnostics/network-check-target-qghnn","kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6rxgf","openshift-multus/multus-q67sd","openshift-ovn-kubernetes/ovnkube-node-z8p6m","kube-system/global-pull-secret-syncer-4w9sx"] Nov 01 21:44:14.124580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124280 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.124580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124284 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 21:44:14.124580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124421 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.124580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124458 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.124580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124536 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.124934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124738 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 21:44:14.124934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.124888 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 21:44:14.125066 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125003 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125158 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125181 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125481 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125700 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125811 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.125886 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126099 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126251 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 21:44:14.126497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126373 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126525 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126645 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126796 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126883 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126882 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.126931 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\"" Nov 01 21:44:14.127020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127008 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\"" Nov 01 21:44:14.127336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127319 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\"" Nov 01 21:44:14.127498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127479 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\"" Nov 01 21:44:14.127579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127530 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\"" Nov 01 21:44:14.127579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127542 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 21:44:14.127579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127542 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 21:44:14.127579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.127572 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\"" Nov 01 21:44:14.128238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128222 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 21:44:14.128334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128317 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.128391 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128321 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.128391 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128379 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 21:44:14.128504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128421 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\"" Nov 01 21:44:14.128504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128442 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\"" Nov 01 21:44:14.128935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128919 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\"" Nov 01 21:44:14.128935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128925 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\"" Nov 01 21:44:14.129095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.128963 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 21:44:14.129095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.129075 4395 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 21:44:14.129095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.129086 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 21:44:14.129204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.129102 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 21:44:14.129204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.129134 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 21:44:14.129204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.129174 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 21:44:14.135916 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.135898 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.135965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.135926 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.135965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.135943 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.136069 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.135968 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.136069 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136022 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.136167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136154 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.136201 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136185 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136202 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136228 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.136313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136243 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136290 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136305 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.136420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136326 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136507 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136488 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z9fn\" (UniqueName: \"kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.136546 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136532 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trdc\" (UniqueName: \"kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.136581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136549 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-multus-daemon-config\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.136618 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136599 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.136701 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136688 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.136792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136781 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.136836 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.136871 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136839 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8vx\" (UniqueName: \"kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.136871 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-config\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.136935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136880 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-tmp\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.136935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136882 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137000 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136973 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137000 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.136992 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137008 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwrg\" (UniqueName: \"kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.137088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137064 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137158 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137092 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137158 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137115 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.137158 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137137 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137167 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137195 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137224 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:14.137338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137320 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137328 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-env-overrides\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137363 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.137490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137473 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137526 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137505 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137526 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovnkube-script-lib\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137538 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:14.137597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137559 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137574 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137590 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137720 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137608 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.137720 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137626 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137720 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137661 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137812 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137765 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0464581-54a9-4673-a69c-3b560089a3c4-etc-tuned\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.137812 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137776 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.137812 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137782 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed118150-74ab-4355-8f26-22d188e0140b-cni-binary-copy\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.137906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137807 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137834 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-tmp-dir\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.137906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137848 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.137906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137880 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.137906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137899 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138066 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137916 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:14.138066 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137940 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138066 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.137976 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h25l\" (UniqueName: \"kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138066 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138043 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.138196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138158 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.138228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138193 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c83a4e94-9b7d-435a-bd02-b043476da87a-metrics-certs\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:14.138228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138223 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.138294 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138267 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.138330 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138301 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.138330 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138307 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6eaffc4a-a597-4820-8a11-31e92606f9f2-original-pull-secret\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.138396 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138402 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.138468 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138428 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.138468 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138453 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfqs\" (UniqueName: \"kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138468 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138459 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-serviceca\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.138558 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138488 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138558 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138506 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138558 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138519 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.138558 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138533 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138597 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138623 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138657 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.138808 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138722 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.138808 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138757 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.138808 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138799 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b809a87-86be-4c57-a0e5-e491568d34c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.138902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138820 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.138902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138842 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.138902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138880 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.138992 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138906 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.138991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139006 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139039 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sww5\" (UniqueName: \"kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139076 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.139193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139092 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1177d05-06de-4967-b190-b00451f8f853-iptables-alerter-script\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.139193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139133 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.139281 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139182 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:44:14.139281 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139242 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:44:14.139281 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139253 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-ovn-node-metrics-cert\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139281 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139275 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.139402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139299 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.139402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139315 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.139402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139389 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ff79f3e-c818-4103-b626-3bed7d7b46b4-konnectivity-ca\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:44:14.139497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.139413 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ff79f3e-c818-4103-b626-3bed7d7b46b4-agent-certs\") pod \"konnectivity-agent-7wgdj\" (UID: \"5ff79f3e-c818-4103-b626-3bed7d7b46b4\") " pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:44:14.146420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.146396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z9fn\" (UniqueName: \"kubernetes.io/projected/d0464581-54a9-4673-a69c-3b560089a3c4-kube-api-access-7z9fn\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.146521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.146505 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8vx\" (UniqueName: \"kubernetes.io/projected/b1177d05-06de-4967-b190-b00451f8f853-kube-api-access-9t8vx\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.146795 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.146777 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trdc\" (UniqueName: \"kubernetes.io/projected/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-kube-api-access-4trdc\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.146836 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.146780 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/c83a4e94-9b7d-435a-bd02-b043476da87a-kube-api-access-66wr8\") pod \"network-metrics-daemon-qs8p9\" (UID: \"c83a4e94-9b7d-435a-bd02-b043476da87a\") " pod="openshift-multus/network-metrics-daemon-qs8p9" Nov 01 21:44:14.146883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.146869 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwrg\" (UniqueName: \"kubernetes.io/projected/0fe1ba52-ea23-43ce-9286-975b58f3998d-kube-api-access-cpwrg\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.147206 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.147186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6d62\" (UniqueName: \"kubernetes.io/projected/a0158144-5a21-4098-869a-dd1598f3aad8-kube-api-access-v6d62\") pod \"network-check-target-qghnn\" (UID: \"a0158144-5a21-4098-869a-dd1598f3aad8\") " pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:14.147505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.147490 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h25l\" (UniqueName: \"kubernetes.io/projected/ed118150-74ab-4355-8f26-22d188e0140b-kube-api-access-5h25l\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.149838 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.149824 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-kube-api-access-dpxpg\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.151829 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.151806 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sww5\" (UniqueName: \"kubernetes.io/projected/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-kube-api-access-8sww5\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.151898 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.151866 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfqs\" (UniqueName: \"kubernetes.io/projected/3b809a87-86be-4c57-a0e5-e491568d34c1-kube-api-access-rqfqs\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.190889 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.190863 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:14.191010 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.190996 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:14.203814 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.203793 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:14.203899 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:14.203834 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:14.204366 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.204348 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 21:44:14.204436 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:14.204410 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-47.us-east-2.compute.internal" Nov 01 21:44:14.239993 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.239958 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.239993 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.239997 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240018 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240081 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240118 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240126 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0-hosts-file\") pod \"node-resolver-mc686\" (UID: \"eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0\") " pod="openshift-dns/node-resolver-mc686" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240165 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240172 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240173 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-lib-modules\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240196 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240211 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-multus-certs\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240233 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240187 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240246 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-kubelet\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240247 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-dbus\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240272 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240291 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240317 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240324 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240327 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-log-socket\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240344 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysctl-conf\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240352 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-bin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240365 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240388 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240403 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240370 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-systemd\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240421 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240439 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-ovn\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240443 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-systemd-units\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240460 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240473 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240477 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-cnibin\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240512 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240518 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240482 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-registration-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240546 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240577 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240622 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-device-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240632 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-kubernetes\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240625 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240637 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6eaffc4a-a597-4820-8a11-31e92606f9f2-kubelet-config\") pod \"global-pull-secret-syncer-4w9sx\" (UID: \"6eaffc4a-a597-4820-8a11-31e92606f9f2\") " pod="kube-system/global-pull-secret-syncer-4w9sx" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240669 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-socket-dir-parent\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240667 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-sysconfig\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240677 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.240704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240718 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240757 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240770 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-sys-fs\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240782 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240808 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240814 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-k8s-cni-cncf-io\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240808 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240838 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-kubelet\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240842 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-host\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240757 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-multus-conf-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240848 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240876 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-run-systemd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240877 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240903 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-var-lib-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240905 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240931 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240935 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-etc-openvswitch\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240954 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240974 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240989 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.240992 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-netd\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241002 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241022 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241068 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241071 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-cnibin\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241068 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-sys\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241091 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-slash\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241103 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-socket-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241123 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241125 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-var-lib-kubelet\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241140 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241169 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241184 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241188 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1177d05-06de-4967-b190-b00451f8f853-host-slash\") pod \"iptables-alerter-c2ntz\" (UID: \"b1177d05-06de-4967-b190-b00451f8f853\") " pod="openshift-network-operator/iptables-alerter-c2ntz" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241200 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241207 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b809a87-86be-4c57-a0e5-e491568d34c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rxgf\" (UID: \"3b809a87-86be-4c57-a0e5-e491568d34c1\") " pod="openshift-multus/multus-additional-cni-plugins-6rxgf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241219 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-cni-bin\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241233 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241237 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-var-lib-cni-multus\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241236 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5-host\") pod \"node-ca-gbcqn\" (UID: \"4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5\") " pod="openshift-image-registry/node-ca-gbcqn" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241248 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241265 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-etc-kubernetes\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241264 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241273 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-run\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241283 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241290 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-system-cni-dir\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241311 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241340 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-hostroot\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241378 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe1ba52-ea23-43ce-9286-975b58f3998d-etc-selinux\") pod \"aws-ebs-csi-driver-node-kxzpg\" (UID: \"0fe1ba52-ea23-43ce-9286-975b58f3998d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kxzpg" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241434 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241456 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241472 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241485 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0464581-54a9-4673-a69c-3b560089a3c4-etc-modprobe-d\") pod \"tuned-nf8vf\" (UID: \"d0464581-54a9-4673-a69c-3b560089a3c4\") " pod="openshift-cluster-node-tuning-operator/tuned-nf8vf" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241509 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-node-log\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241511 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed118150-74ab-4355-8f26-22d188e0140b-host-run-netns\") pod \"multus-q67sd\" (UID: \"ed118150-74ab-4355-8f26-22d188e0140b\") " pod="openshift-multus/multus-q67sd" Nov 01 21:44:14.241943 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.241541 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac-host-run-netns\") pod \"ovnkube-node-z8p6m\" (UID: \"51f71dcf-17d5-4fb3-ad84-c83bf8fb40ac\") " pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:14.759716 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:14.759681 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:15.018336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:15.018261 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:15.421215 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:15.421134 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:15.885919 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:15.885889 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z8p6m" Nov 01 21:44:21.482129 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:21.482098 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7wgdj" Nov 01 21:44:23.122412 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.122345 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t4vbj"] Nov 01 21:44:23.125528 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.125478 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp"] Nov 01 21:44:23.125671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.125614 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.128256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.128239 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:44:23.128379 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.128362 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:23.130750 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.130731 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9fvgv"] Nov 01 21:44:23.130855 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.130839 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.135116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.135096 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.140985 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.140968 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 21:44:23.141134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.141118 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 21:44:23.141244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.141225 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 21:44:23.141481 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.141467 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 21:44:23.141567 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.141553 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 21:44:23.141627 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.141565 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 21:44:23.142315 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.142282 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 21:44:23.142596 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.142331 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 21:44:23.143331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.143314 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zq7fh\"" Nov 01 21:44:23.143814 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.143797 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4vbj"] Nov 01 21:44:23.146086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.146040 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp"] Nov 01 21:44:23.147775 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.147721 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9fvgv"] Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.148600 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.148877 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.148900 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.148925 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.148989 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-5jk42\"" Nov 01 21:44:23.149414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.149085 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pth5p\"" Nov 01 21:44:23.150298 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.150278 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:44:23.150862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.150843 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-47fcd\"" Nov 01 21:44:23.160643 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.160626 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bj7ms"] Nov 01 21:44:23.163274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.163258 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.173221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.173199 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bj7ms"] Nov 01 21:44:23.173322 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.173279 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 21:44:23.175727 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.175710 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 21:44:23.186140 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186120 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60f2807-1f57-4879-91c6-21d9b47c3917-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.186225 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186146 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb80e501-2a27-4977-ae29-3804022b6d13-cert\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.186225 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186165 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.186225 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186183 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qrz\" (UniqueName: \"kubernetes.io/projected/bb80e501-2a27-4977-ae29-3804022b6d13-kube-api-access-j6qrz\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.186225 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186199 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4874\" (UniqueName: \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186314 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2q6\" (UniqueName: \"kubernetes.io/projected/78773b68-9492-41e9-948b-832944834e9e-kube-api-access-wx2q6\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.186353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186330 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186345 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78773b68-9492-41e9-948b-832944834e9e-tmp\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.186442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186360 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186376 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl2h\" (UniqueName: \"kubernetes.io/projected/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-api-access-lcl2h\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.186442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186412 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186529 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186443 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186529 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186508 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186553 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60f2807-1f57-4879-91c6-21d9b47c3917-data-volume\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.186591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186573 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14d7ca00-d8c0-4916-b9f3-1fe77186a182-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nbjcp\" (UID: \"14d7ca00-d8c0-4916-b9f3-1fe77186a182\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:23.186650 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186589 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.186650 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.186605 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60f2807-1f57-4879-91c6-21d9b47c3917-crio-socket\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.195521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.195495 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rqr8c"] Nov 01 21:44:23.198410 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.198392 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.201919 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.201902 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv"] Nov 01 21:44:23.204664 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.204634 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.211954 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.211937 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 21:44:23.219991 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.219267 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 21:44:23.219991 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.219395 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rqr8c"] Nov 01 21:44:23.234804 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.234779 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv"] Nov 01 21:44:23.245542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245520 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 21:44:23.245658 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245563 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pn6l\"" Nov 01 21:44:23.245751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245735 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fpfrj\"" Nov 01 21:44:23.245816 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245741 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 21:44:23.245816 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245756 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 21:44:23.245911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245852 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 21:44:23.245911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.245897 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 21:44:23.246049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.246017 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bzsh4\"" Nov 01 21:44:23.286899 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.286874 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-tls\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287124 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.286928 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60f2807-1f57-4879-91c6-21d9b47c3917-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.287124 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.286974 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qrz\" (UniqueName: \"kubernetes.io/projected/bb80e501-2a27-4977-ae29-3804022b6d13-kube-api-access-j6qrz\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.287320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287298 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.287373 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287355 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.287429 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287379 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2q6\" (UniqueName: \"kubernetes.io/projected/78773b68-9492-41e9-948b-832944834e9e-kube-api-access-wx2q6\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.287480 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287414 4395 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 21:44:23.287480 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287464 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ltk\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-kube-api-access-p2ltk\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287502 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78773b68-9492-41e9-948b-832944834e9e-tmp\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.287582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287531 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.287679 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287616 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-trusted-ca\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287733 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287679 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-installation-pull-secrets\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287733 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287716 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-certificates\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287837 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287757 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-bound-sa-token\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.287837 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287792 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.287937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287794 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-config-volume\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.287937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287917 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-metrics-tls\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.287937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287927 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78773b68-9492-41e9-948b-832944834e9e-tmp\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.288083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287952 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1e5511f-afdf-4afd-850e-ba9171fcc326-ca-trust-extracted\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.288083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287956 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.288083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.287999 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14d7ca00-d8c0-4916-b9f3-1fe77186a182-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nbjcp\" (UID: \"14d7ca00-d8c0-4916-b9f3-1fe77186a182\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:23.288202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288123 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288159 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60f2807-1f57-4879-91c6-21d9b47c3917-crio-socket\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.288202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288189 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-tmp-dir\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.288337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288218 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb80e501-2a27-4977-ae29-3804022b6d13-cert\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.288337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288248 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60f2807-1f57-4879-91c6-21d9b47c3917-crio-socket\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.288337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288255 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4874\" (UniqueName: \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288323 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-image-registry-private-configuration\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.288506 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288467 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288560 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288501 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl2h\" (UniqueName: \"kubernetes.io/projected/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-api-access-lcl2h\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.288560 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288528 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288557 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288591 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnh7r\" (UniqueName: \"kubernetes.io/projected/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-kube-api-access-vnh7r\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.288657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288616 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60f2807-1f57-4879-91c6-21d9b47c3917-data-volume\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.288880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288827 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.288982 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.288900 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60f2807-1f57-4879-91c6-21d9b47c3917-data-volume\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.289551 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.289502 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.289776 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.289750 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.290359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.290339 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60f2807-1f57-4879-91c6-21d9b47c3917-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.290592 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.290572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.291045 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.291011 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb80e501-2a27-4977-ae29-3804022b6d13-cert\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.291149 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.291131 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14d7ca00-d8c0-4916-b9f3-1fe77186a182-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nbjcp\" (UID: \"14d7ca00-d8c0-4916-b9f3-1fe77186a182\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:23.291451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.291431 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.300809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.300784 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qrz\" (UniqueName: \"kubernetes.io/projected/bb80e501-2a27-4977-ae29-3804022b6d13-kube-api-access-j6qrz\") pod \"ingress-canary-bj7ms\" (UID: \"bb80e501-2a27-4977-ae29-3804022b6d13\") " pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.301276 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.301256 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl2h\" (UniqueName: \"kubernetes.io/projected/d60f2807-1f57-4879-91c6-21d9b47c3917-kube-api-access-lcl2h\") pod \"insights-runtime-extractor-t4vbj\" (UID: \"d60f2807-1f57-4879-91c6-21d9b47c3917\") " pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.301988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.301971 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4874\" (UniqueName: \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\") pod \"console-7b5756946f-zkqm5\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.302256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.302238 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2q6\" (UniqueName: \"kubernetes.io/projected/78773b68-9492-41e9-948b-832944834e9e-kube-api-access-wx2q6\") pod \"downloads-6dd789c8b7-9fvgv\" (UID: \"78773b68-9492-41e9-948b-832944834e9e\") " pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.389612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389538 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-image-registry-private-configuration\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.389612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389589 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnh7r\" (UniqueName: \"kubernetes.io/projected/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-kube-api-access-vnh7r\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.389820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389621 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-tls\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.389820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389662 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ltk\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-kube-api-access-p2ltk\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.389820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389694 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-trusted-ca\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.389820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389763 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-installation-pull-secrets\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.389820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-certificates\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.390076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389829 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-bound-sa-token\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.390076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389861 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-config-volume\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.390076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389885 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-metrics-tls\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.390076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389909 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1e5511f-afdf-4afd-850e-ba9171fcc326-ca-trust-extracted\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.390076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.389945 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-tmp-dir\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.390895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.390707 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-config-volume\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.390895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.390834 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1e5511f-afdf-4afd-850e-ba9171fcc326-ca-trust-extracted\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.390895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.390876 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-certificates\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.390895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.390888 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1e5511f-afdf-4afd-850e-ba9171fcc326-trusted-ca\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.391188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.391103 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-tmp-dir\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.392428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.392404 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-image-registry-private-configuration\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.392540 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.392467 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1e5511f-afdf-4afd-850e-ba9171fcc326-installation-pull-secrets\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.393138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.393121 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-metrics-tls\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.393404 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.393385 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-registry-tls\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.403819 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.403798 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnh7r\" (UniqueName: \"kubernetes.io/projected/6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4-kube-api-access-vnh7r\") pod \"dns-default-rqr8c\" (UID: \"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4\") " pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.403950 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.403933 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ltk\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-kube-api-access-p2ltk\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.404125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.404111 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1e5511f-afdf-4afd-850e-ba9171fcc326-bound-sa-token\") pod \"image-registry-7b7d4c5ff6-hs6vv\" (UID: \"a1e5511f-afdf-4afd-850e-ba9171fcc326\") " pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.445077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.445058 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4vbj" Nov 01 21:44:23.448198 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.448180 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:23.456316 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.456299 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:23.462310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.462291 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:23.473119 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.473093 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bj7ms" Nov 01 21:44:23.512525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.512177 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:23.524433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.521629 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:23.729279 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.729211 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4vbj"] Nov 01 21:44:23.732567 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.732539 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60f2807_1f57_4879_91c6_21d9b47c3917.slice/crio-e67ce5130b59d6e0212d08d1e6f6b4ad8d0c6c7bf98a859a008613dfb98eea72 WatchSource:0}: Error finding container e67ce5130b59d6e0212d08d1e6f6b4ad8d0c6c7bf98a859a008613dfb98eea72: Status 404 returned error can't find the container with id e67ce5130b59d6e0212d08d1e6f6b4ad8d0c6c7bf98a859a008613dfb98eea72 Nov 01 21:44:23.745386 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.745366 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp"] Nov 01 21:44:23.748193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.748165 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:44:23.748423 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.748397 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d7ca00_d8c0_4916_b9f3_1fe77186a182.slice/crio-b9b56d57c3b9fbf9644e6389bf010e4853fafb802515667686d15a3c79fe7056 WatchSource:0}: Error finding container b9b56d57c3b9fbf9644e6389bf010e4853fafb802515667686d15a3c79fe7056: Status 404 returned error can't find the container with id b9b56d57c3b9fbf9644e6389bf010e4853fafb802515667686d15a3c79fe7056 Nov 01 21:44:23.750264 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.750248 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:44:23.767944 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.767917 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d50a78c_2d18_48d2_9ef3_ea388ba46ffb.slice/crio-e5340dcc86716732a0cfc8eb66b907146e22e0bf512d57bde500dda64fc55150 WatchSource:0}: Error finding container e5340dcc86716732a0cfc8eb66b907146e22e0bf512d57bde500dda64fc55150: Status 404 returned error can't find the container with id e5340dcc86716732a0cfc8eb66b907146e22e0bf512d57bde500dda64fc55150 Nov 01 21:44:23.778081 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.778061 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bj7ms"] Nov 01 21:44:23.778127 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.778109 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9fvgv"] Nov 01 21:44:23.784883 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.784859 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb80e501_2a27_4977_ae29_3804022b6d13.slice/crio-0e02efb6c02bcaa89aa933a13d74c3e1497a9b743ab65626eafadc3f088f5309 WatchSource:0}: Error finding container 0e02efb6c02bcaa89aa933a13d74c3e1497a9b743ab65626eafadc3f088f5309: Status 404 returned error can't find the container with id 0e02efb6c02bcaa89aa933a13d74c3e1497a9b743ab65626eafadc3f088f5309 Nov 01 21:44:23.785205 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.785184 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78773b68_9492_41e9_948b_832944834e9e.slice/crio-dfb47ed2747491ef7e00bacac8b388de67a248750e1f0058ba6ddc6afec7e2c8 WatchSource:0}: Error finding container dfb47ed2747491ef7e00bacac8b388de67a248750e1f0058ba6ddc6afec7e2c8: Status 404 returned error can't find the container with id dfb47ed2747491ef7e00bacac8b388de67a248750e1f0058ba6ddc6afec7e2c8 Nov 01 21:44:23.845314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.845272 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rqr8c"] Nov 01 21:44:23.848499 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.848473 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca5f5bc_7dfd_4b19_ae0c_2d654538e1b4.slice/crio-69ce33cd243034ad38462f733bb0a894814cd33788d55f205ed6daea3d858062 WatchSource:0}: Error finding container 69ce33cd243034ad38462f733bb0a894814cd33788d55f205ed6daea3d858062: Status 404 returned error can't find the container with id 69ce33cd243034ad38462f733bb0a894814cd33788d55f205ed6daea3d858062 Nov 01 21:44:23.852842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:23.852820 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv"] Nov 01 21:44:23.856002 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:23.855981 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e5511f_afdf_4afd_850e_ba9171fcc326.slice/crio-ce03a42dd8337a492255182dc2f879717ed5ab55e08b141c128cb1f3437a0748 WatchSource:0}: Error finding container ce03a42dd8337a492255182dc2f879717ed5ab55e08b141c128cb1f3437a0748: Status 404 returned error can't find the container with id ce03a42dd8337a492255182dc2f879717ed5ab55e08b141c128cb1f3437a0748 Nov 01 21:44:24.214824 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.214791 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9fvgv" event={"ID":"78773b68-9492-41e9-948b-832944834e9e","Type":"ContainerStarted","Data":"dfb47ed2747491ef7e00bacac8b388de67a248750e1f0058ba6ddc6afec7e2c8"} Nov 01 21:44:24.216527 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.216496 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4vbj" event={"ID":"d60f2807-1f57-4879-91c6-21d9b47c3917","Type":"ContainerStarted","Data":"2a0e13de3ecaf00d4d3421111bda71dd17ef43b2750705a9a46e285acc9f86af"} Nov 01 21:44:24.216655 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.216562 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4vbj" event={"ID":"d60f2807-1f57-4879-91c6-21d9b47c3917","Type":"ContainerStarted","Data":"e67ce5130b59d6e0212d08d1e6f6b4ad8d0c6c7bf98a859a008613dfb98eea72"} Nov 01 21:44:24.217869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.217845 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5756946f-zkqm5" event={"ID":"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb","Type":"ContainerStarted","Data":"e5340dcc86716732a0cfc8eb66b907146e22e0bf512d57bde500dda64fc55150"} Nov 01 21:44:24.219421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.219392 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" event={"ID":"a1e5511f-afdf-4afd-850e-ba9171fcc326","Type":"ContainerStarted","Data":"859dced25c3658501907e048d5294caccad02eeacdb6088c61ef63f03d1393c6"} Nov 01 21:44:24.219526 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.219433 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" event={"ID":"a1e5511f-afdf-4afd-850e-ba9171fcc326","Type":"ContainerStarted","Data":"ce03a42dd8337a492255182dc2f879717ed5ab55e08b141c128cb1f3437a0748"} Nov 01 21:44:24.219600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.219586 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:24.220564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.220534 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rqr8c" event={"ID":"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4","Type":"ContainerStarted","Data":"69ce33cd243034ad38462f733bb0a894814cd33788d55f205ed6daea3d858062"} Nov 01 21:44:24.221780 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.221762 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" event={"ID":"14d7ca00-d8c0-4916-b9f3-1fe77186a182","Type":"ContainerStarted","Data":"b9b56d57c3b9fbf9644e6389bf010e4853fafb802515667686d15a3c79fe7056"} Nov 01 21:44:24.222909 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.222888 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bj7ms" event={"ID":"bb80e501-2a27-4977-ae29-3804022b6d13","Type":"ContainerStarted","Data":"0e02efb6c02bcaa89aa933a13d74c3e1497a9b743ab65626eafadc3f088f5309"} Nov 01 21:44:24.242828 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:24.242747 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" podStartSLOduration=1.242727703 podStartE2EDuration="1.242727703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:24.242226811 +0000 UTC m=+11.216129351" watchObservedRunningTime="2026-11-01 21:44:24.242727703 +0000 UTC m=+11.216630242" Nov 01 21:44:25.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:25.235473 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4vbj" event={"ID":"d60f2807-1f57-4879-91c6-21d9b47c3917","Type":"ContainerStarted","Data":"db9233f6e8af94fdaaf2f2b41f31bfa5354e9c8ba5f05b30d507527752e4cec1"} Nov 01 21:44:26.242810 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:26.242767 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" event={"ID":"14d7ca00-d8c0-4916-b9f3-1fe77186a182","Type":"ContainerStarted","Data":"20654e53be1d8f2cafdc87684753daa92ab1fdf32134db8b40f5b661df37d021"} Nov 01 21:44:26.244120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:26.243998 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:26.249409 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:26.249376 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" Nov 01 21:44:26.280267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:26.280102 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nbjcp" podStartSLOduration=1.63192747 podStartE2EDuration="3.280082467s" totalImagesPullingTime="1.648154997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:26.262410962 +0000 UTC m=+13.236313502" watchObservedRunningTime="2026-11-01 21:44:26.280082467 +0000 UTC m=+13.253985005" Nov 01 21:44:27.045205 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.045170 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hl584"] Nov 01 21:44:27.048799 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.048776 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.053140 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.053094 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 21:44:27.054859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.054833 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 21:44:27.054982 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.054896 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 21:44:27.054982 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.054969 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vg2rp\"" Nov 01 21:44:27.055136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.055071 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 21:44:27.055226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.055209 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 21:44:27.066324 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.065309 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hl584"] Nov 01 21:44:27.127043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.126978 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.127219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.127055 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.127219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.127148 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxx5c\" (UniqueName: \"kubernetes.io/projected/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-kube-api-access-pxx5c\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.127219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.127194 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.228373 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.228337 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.228538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.228382 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.228538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.228402 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxx5c\" (UniqueName: \"kubernetes.io/projected/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-kube-api-access-pxx5c\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.228538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.228420 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.229431 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.229399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.232285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.232233 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.234051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.233217 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.242274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.242256 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxx5c\" (UniqueName: \"kubernetes.io/projected/71383c47-4a77-4bc3-9f3f-2e4cecdbe99f-kube-api-access-pxx5c\") pod \"prometheus-operator-9cd89b74c-hl584\" (UID: \"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.249924 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.249692 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rqr8c" event={"ID":"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4","Type":"ContainerStarted","Data":"ce59469e30a413d83f5346daaeb8486381b2b2c1dc1d215120158a404f96c9f8"} Nov 01 21:44:27.249924 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.249742 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rqr8c" event={"ID":"6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4","Type":"ContainerStarted","Data":"9723fbcdbbd07b66c9f5dc8d18234b8b166f0ca6d9c173e7034c91b9651ea4b6"} Nov 01 21:44:27.249924 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.249794 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:27.268862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.268803 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rqr8c" podStartSLOduration=1.654071557 podStartE2EDuration="4.268786682s" totalImagesPullingTime="2.614715125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:27.268396737 +0000 UTC m=+14.242299282" watchObservedRunningTime="2026-11-01 21:44:27.268786682 +0000 UTC m=+14.242689221" Nov 01 21:44:27.373624 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.373543 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" Nov 01 21:44:27.674274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.674199 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hl584"] Nov 01 21:44:27.681369 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:27.681333 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71383c47_4a77_4bc3_9f3f_2e4cecdbe99f.slice/crio-6d9107fc25ed92575367d01f2ff4095a164a89d9c320a7876ef5edd185f5ab62 WatchSource:0}: Error finding container 6d9107fc25ed92575367d01f2ff4095a164a89d9c320a7876ef5edd185f5ab62: Status 404 returned error can't find the container with id 6d9107fc25ed92575367d01f2ff4095a164a89d9c320a7876ef5edd185f5ab62 Nov 01 21:44:27.805443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.805410 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:44:27.810280 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.810259 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.819086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.819059 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:44:27.825287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.825263 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 21:44:27.934543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934446 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934507 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934604 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934641 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934706 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934734 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934762 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24n6q\" (UniqueName: \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:27.934996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:27.934793 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036242 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036204 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036280 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24n6q\" (UniqueName: \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036311 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036339 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036376 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036421 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.036735 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.036457 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.037055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.037002 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.037158 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.037104 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.037234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.037210 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.037336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.037318 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.037686 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.037616 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.040617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.040599 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.052415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.052391 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24n6q\" (UniqueName: \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.058873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.058850 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\") pod \"console-686fb85f67-vvzm9\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.137453 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.137414 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:28.254532 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.253928 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bj7ms" event={"ID":"bb80e501-2a27-4977-ae29-3804022b6d13","Type":"ContainerStarted","Data":"6bb5e6a65268d12bbdd5ecb9bded0fdd238d3509254a7f749a1ba3bf843abc26"} Nov 01 21:44:28.258399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.257122 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4vbj" event={"ID":"d60f2807-1f57-4879-91c6-21d9b47c3917","Type":"ContainerStarted","Data":"f78d4a3784ed4735577c45b429a409120d5ff724990937199e30528b749aeefc"} Nov 01 21:44:28.259665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.259638 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" event={"ID":"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f","Type":"ContainerStarted","Data":"6d9107fc25ed92575367d01f2ff4095a164a89d9c320a7876ef5edd185f5ab62"} Nov 01 21:44:28.272908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.272859 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bj7ms" podStartSLOduration=1.288147309 podStartE2EDuration="5.272844224s" totalImagesPullingTime="3.984696915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:28.272085309 +0000 UTC m=+15.245987884" watchObservedRunningTime="2026-11-01 21:44:28.272844224 +0000 UTC m=+15.246746762" Nov 01 21:44:28.294053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.293918 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t4vbj" podStartSLOduration=1.044367536 podStartE2EDuration="5.293902335s" totalImagesPullingTime="4.249534799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:28.293168898 +0000 UTC m=+15.267071437" watchObservedRunningTime="2026-11-01 21:44:28.293902335 +0000 UTC m=+15.267804875" Nov 01 21:44:28.422788 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:28.422750 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:44:28.426919 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:28.426889 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e367fa1_029f_436c_89c6_093395c89235.slice/crio-dad1d2fa470b98cd8de575f21c73c1c88b407c53516be94e688f81c2b6bf7599 WatchSource:0}: Error finding container dad1d2fa470b98cd8de575f21c73c1c88b407c53516be94e688f81c2b6bf7599: Status 404 returned error can't find the container with id dad1d2fa470b98cd8de575f21c73c1c88b407c53516be94e688f81c2b6bf7599 Nov 01 21:44:29.264676 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:29.264646 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686fb85f67-vvzm9" event={"ID":"5e367fa1-029f-436c-89c6-093395c89235","Type":"ContainerStarted","Data":"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329"} Nov 01 21:44:29.265315 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:29.264701 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686fb85f67-vvzm9" event={"ID":"5e367fa1-029f-436c-89c6-093395c89235","Type":"ContainerStarted","Data":"dad1d2fa470b98cd8de575f21c73c1c88b407c53516be94e688f81c2b6bf7599"} Nov 01 21:44:29.266882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:29.266854 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5756946f-zkqm5" event={"ID":"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb","Type":"ContainerStarted","Data":"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442"} Nov 01 21:44:29.290310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:29.290270 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-686fb85f67-vvzm9" podStartSLOduration=1.8069163879999999 podStartE2EDuration="2.290256429s" totalImagesPullingTime="483.340041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:29.289515693 +0000 UTC m=+16.263418233" watchObservedRunningTime="2026-11-01 21:44:29.290256429 +0000 UTC m=+16.264159021" Nov 01 21:44:29.309714 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:29.309556 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b5756946f-zkqm5" podStartSLOduration=1.166387001 podStartE2EDuration="6.309540486s" totalImagesPullingTime="5.143153485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:29.309249313 +0000 UTC m=+16.283151852" watchObservedRunningTime="2026-11-01 21:44:29.309540486 +0000 UTC m=+16.283443026" Nov 01 21:44:30.272584 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:30.272544 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" event={"ID":"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f","Type":"ContainerStarted","Data":"9cac8b35e911ad65064c76b70db8b3b4d7a210bc2eaa223e0c06605e63db8192"} Nov 01 21:44:30.273070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:30.272610 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" event={"ID":"71383c47-4a77-4bc3-9f3f-2e4cecdbe99f","Type":"ContainerStarted","Data":"62a701b6ab0944e43b4fa06ef83e99579574bb6aa8f2e3f09f157bfa152ee344"} Nov 01 21:44:30.293211 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:30.293144 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hl584" podStartSLOduration=1.721614202 podStartE2EDuration="3.29309487s" totalImagesPullingTime="1.571480668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:30.292185257 +0000 UTC m=+17.266087798" watchObservedRunningTime="2026-11-01 21:44:30.29309487 +0000 UTC m=+17.266997410" Nov 01 21:44:32.420849 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.420815 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-48klz"] Nov 01 21:44:32.424422 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.424400 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.450760 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.450733 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 21:44:32.450907 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.450812 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bpkb5\"" Nov 01 21:44:32.450907 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.450862 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 21:44:32.472253 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472229 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-metrics-client-ca\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472265 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-tls\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472297 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-textfile\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472347 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472368 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472561 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-root\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472628 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472589 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghv2s\" (UniqueName: \"kubernetes.io/projected/3e55adf1-4ef3-49bf-a410-e80a919374f7-kube-api-access-ghv2s\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472628 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472613 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-wtmp\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.472691 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.472661 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-sys\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.522788 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.522758 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 21:44:32.573103 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573077 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-metrics-client-ca\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573108 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-tls\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573129 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-textfile\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573274 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573317 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-root\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573386 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-textfile\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573401 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghv2s\" (UniqueName: \"kubernetes.io/projected/3e55adf1-4ef3-49bf-a410-e80a919374f7-kube-api-access-ghv2s\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-wtmp\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573464 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-root\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573480 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-sys\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573568 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-sys\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573592 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-wtmp\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573717 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-metrics-client-ca\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.573842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.573819 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.575788 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.575768 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.575881 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.575832 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e55adf1-4ef3-49bf-a410-e80a919374f7-node-exporter-tls\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.586641 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.586617 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghv2s\" (UniqueName: \"kubernetes.io/projected/3e55adf1-4ef3-49bf-a410-e80a919374f7-kube-api-access-ghv2s\") pod \"node-exporter-48klz\" (UID: \"3e55adf1-4ef3-49bf-a410-e80a919374f7\") " pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.741399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:32.741330 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48klz" Nov 01 21:44:32.753862 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:32.753821 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e55adf1_4ef3_49bf_a410_e80a919374f7.slice/crio-3878b85ef19bcdbde00b9607bf397a7e7fd204f267d6be0164280c121071a6f5 WatchSource:0}: Error finding container 3878b85ef19bcdbde00b9607bf397a7e7fd204f267d6be0164280c121071a6f5: Status 404 returned error can't find the container with id 3878b85ef19bcdbde00b9607bf397a7e7fd204f267d6be0164280c121071a6f5 Nov 01 21:44:33.284136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:33.284099 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48klz" event={"ID":"3e55adf1-4ef3-49bf-a410-e80a919374f7","Type":"ContainerStarted","Data":"3878b85ef19bcdbde00b9607bf397a7e7fd204f267d6be0164280c121071a6f5"} Nov 01 21:44:33.456514 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:33.456474 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:33.456963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:33.456581 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:33.461658 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:33.461631 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:34.290856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:34.290818 4395 generic.go:356] "Generic (PLEG): container finished" podID="3e55adf1-4ef3-49bf-a410-e80a919374f7" containerID="9c25af0ecc6a11ba048fcc75dcc33ade3743577933b6dd120179141f31ab804a" exitCode=0 Nov 01 21:44:34.291022 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:34.290912 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48klz" event={"ID":"3e55adf1-4ef3-49bf-a410-e80a919374f7","Type":"ContainerDied","Data":"9c25af0ecc6a11ba048fcc75dcc33ade3743577933b6dd120179141f31ab804a"} Nov 01 21:44:34.295680 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:34.295655 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:44:35.297451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:35.297417 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48klz" event={"ID":"3e55adf1-4ef3-49bf-a410-e80a919374f7","Type":"ContainerStarted","Data":"26776d8807d7920da60b938f081284d925118b602867e992a6fdab2d2c7747c3"} Nov 01 21:44:35.297892 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:35.297476 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48klz" event={"ID":"3e55adf1-4ef3-49bf-a410-e80a919374f7","Type":"ContainerStarted","Data":"d43c7efa3bde3cfd17e84d75e3020880053489dd16b3d48fa675900f4cdd91e0"} Nov 01 21:44:35.321958 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:35.321914 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-48klz" podStartSLOduration=2.16231738 podStartE2EDuration="3.321899592s" totalImagesPullingTime="1.085298249s" totalInitContainerRuntime="74.283963ms" isStatefulPod=true podCreationTimestamp="2026-11-01 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:35.320204459 +0000 UTC m=+22.294106991" watchObservedRunningTime="2026-11-01 21:44:35.321899592 +0000 UTC m=+22.295802130" Nov 01 21:44:38.138043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.137990 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:38.138532 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.138096 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:38.143087 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.143018 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:38.269201 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.269171 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rqr8c" Nov 01 21:44:38.315559 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.315529 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:44:38.370195 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:38.370162 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:44:40.610915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.610884 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:44:40.615245 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.615223 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.625336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.625274 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:44:40.757956 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.757928 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.757956 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.757959 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758094 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758126 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758150 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758258 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvpp\" (UniqueName: \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758337 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758319 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.758376 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.758354 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859284 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859253 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859284 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859286 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859300 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859320 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvpp\" (UniqueName: \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859360 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859380 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859410 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.859511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.859429 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.860440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.860402 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.860559 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.860467 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.860977 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.860928 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.861201 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.861178 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.861351 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.861331 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.862300 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.862271 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.863621 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.863601 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.872970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.872952 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvpp\" (UniqueName: \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\") pod \"console-6555c7dbc8-8fzbt\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:40.929042 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:40.928996 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:41.215470 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:41.215380 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:44:41.222209 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:44:41.222179 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd89f58_f3b8_447e_99a5_0c0f2ff479aa.slice/crio-b8a4062306dbc697646f836c134972ce2924e9ea8b1ef4c163b681f414751631 WatchSource:0}: Error finding container b8a4062306dbc697646f836c134972ce2924e9ea8b1ef4c163b681f414751631: Status 404 returned error can't find the container with id b8a4062306dbc697646f836c134972ce2924e9ea8b1ef4c163b681f414751631 Nov 01 21:44:41.324911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:41.324877 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6555c7dbc8-8fzbt" event={"ID":"acd89f58-f3b8-447e-99a5-0c0f2ff479aa","Type":"ContainerStarted","Data":"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c"} Nov 01 21:44:41.325101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:41.324936 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6555c7dbc8-8fzbt" event={"ID":"acd89f58-f3b8-447e-99a5-0c0f2ff479aa","Type":"ContainerStarted","Data":"b8a4062306dbc697646f836c134972ce2924e9ea8b1ef4c163b681f414751631"} Nov 01 21:44:41.346812 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:41.346753 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6555c7dbc8-8fzbt" podStartSLOduration=1.346732946 podStartE2EDuration="1.346732946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:41.346213362 +0000 UTC m=+28.320115901" watchObservedRunningTime="2026-11-01 21:44:41.346732946 +0000 UTC m=+28.320635483" Nov 01 21:44:44.345182 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:44.345137 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9fvgv" event={"ID":"78773b68-9492-41e9-948b-832944834e9e","Type":"ContainerStarted","Data":"0105edc9df2c73793de6d58ce9d31023f0c6ba25db88566e118564ee8f3696db"} Nov 01 21:44:44.345667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:44.345386 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:44.347022 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:44.346995 4395 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-9fvgv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" start-of-body= Nov 01 21:44:44.347152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:44.347070 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-9fvgv" podUID="78773b68-9492-41e9-948b-832944834e9e" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" Nov 01 21:44:44.366329 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:44.366281 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-9fvgv" podStartSLOduration=1.175292022 podStartE2EDuration="21.366267238s" totalImagesPullingTime="20.190975216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:44.36528227 +0000 UTC m=+31.339184811" watchObservedRunningTime="2026-11-01 21:44:44.366267238 +0000 UTC m=+31.340169831" Nov 01 21:44:45.196557 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:45.196527 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qghnn" Nov 01 21:44:45.241279 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:45.241248 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b7d4c5ff6-hs6vv" Nov 01 21:44:45.361747 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:45.361711 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9fvgv" Nov 01 21:44:50.929491 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:50.929451 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:50.930097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:50.929561 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:50.934585 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:50.934561 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:51.375379 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:51.375345 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:44:51.424749 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:44:51.424717 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:44:52.307223 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:44:52.307188 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:45:03.395565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.395456 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b5756946f-zkqm5" podUID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" containerName="console" containerID="cri-o://ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442" gracePeriod=15 Nov 01 21:45:03.674964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.674908 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:45:03.755684 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755659 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.755803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755695 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\" (UniqueName: \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.755803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755718 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.755803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755751 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.755803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755785 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.756056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.756056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.755872 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\") pod \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\" (UID: \"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb\") " Nov 01 21:45:03.756169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.756122 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:03.756228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.756163 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:45:03.756228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.756185 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:03.756405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.756353 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:03.758080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.758018 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:03.758080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.758058 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "kube-api-access-b4874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:03.758228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.758178 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert" pod "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" (UID: "2d50a78c-2d18-48d2-9ef3-ea388ba46ffb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:03.857141 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857115 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857141 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857136 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-tmp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857146 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-oauth-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857154 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857163 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4874\" (UniqueName: \"kubernetes.io/projected/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-kube-api-access-b4874\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857172 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:03.857267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:03.857181 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb-console-oauth-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:04.416994 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.416959 4395 generic.go:356] "Generic (PLEG): container finished" podID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" containerID="ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442" exitCode=0 Nov 01 21:45:04.417490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.417015 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5756946f-zkqm5" Nov 01 21:45:04.417490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.417055 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5756946f-zkqm5" event={"ID":"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb","Type":"ContainerDied","Data":"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442"} Nov 01 21:45:04.417490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.417107 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5756946f-zkqm5" event={"ID":"2d50a78c-2d18-48d2-9ef3-ea388ba46ffb","Type":"ContainerDied","Data":"e5340dcc86716732a0cfc8eb66b907146e22e0bf512d57bde500dda64fc55150"} Nov 01 21:45:04.417490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.417128 4395 scope.go:122] "RemoveContainer" containerID="ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442" Nov 01 21:45:04.425305 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.425289 4395 scope.go:122] "RemoveContainer" containerID="ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442" Nov 01 21:45:04.425593 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:45:04.425574 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442\": container with ID starting with ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442 not found: ID does not exist" containerID="ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442" Nov 01 21:45:04.425652 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.425602 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442"} err="failed to get container status \"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442\": rpc error: code = NotFound desc = could not find container \"ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442\": container with ID starting with ba1a2fe6d45366887e0f30731af19237dd67ebcfe9744c28a501b0ab78c78442 not found: ID does not exist" Nov 01 21:45:04.437474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.437450 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:45:04.441840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:04.441819 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b5756946f-zkqm5"] Nov 01 21:45:05.161710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:05.161680 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" path="/var/lib/kubelet/pods/2d50a78c-2d18-48d2-9ef3-ea388ba46ffb/volumes" Nov 01 21:45:16.448534 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.448472 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-686fb85f67-vvzm9" podUID="5e367fa1-029f-436c-89c6-093395c89235" containerName="console" containerID="cri-o://73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329" gracePeriod=15 Nov 01 21:45:16.719858 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.719806 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:45:16.743215 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743185 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743342 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743235 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743342 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743271 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743342 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743324 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743355 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743394 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743424 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\" (UniqueName: \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743470 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\") pod \"5e367fa1-029f-436c-89c6-093395c89235\" (UID: \"5e367fa1-029f-436c-89c6-093395c89235\") " Nov 01 21:45:16.743776 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743519 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:16.743776 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743682 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.743984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.743912 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:16.744697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.744455 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:45:16.744697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.744532 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:16.744697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.744552 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:16.746095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.746053 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "kube-api-access-24n6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:16.746487 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.746466 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:16.746689 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.746674 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config" pod "5e367fa1-029f-436c-89c6-093395c89235" (UID: "5e367fa1-029f-436c-89c6-093395c89235"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:16.820908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.820879 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4"] Nov 01 21:45:16.821268 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821251 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" containerName="console" Nov 01 21:45:16.821313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821273 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" containerName="console" Nov 01 21:45:16.821313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821308 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5e367fa1-029f-436c-89c6-093395c89235" containerName="console" Nov 01 21:45:16.821378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821316 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e367fa1-029f-436c-89c6-093395c89235" containerName="console" Nov 01 21:45:16.821409 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821379 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e367fa1-029f-436c-89c6-093395c89235" containerName="console" Nov 01 21:45:16.821409 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.821392 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d50a78c-2d18-48d2-9ef3-ea388ba46ffb" containerName="console" Nov 01 21:45:16.844334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844306 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-oauth-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844338 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e367fa1-029f-436c-89c6-093395c89235-console-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844354 4395 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-trusted-ca-bundle\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844369 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-oauth-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844384 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e367fa1-029f-436c-89c6-093395c89235-console-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844398 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24n6q\" (UniqueName: \"kubernetes.io/projected/5e367fa1-029f-436c-89c6-093395c89235-kube-api-access-24n6q\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.844441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.844413 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e367fa1-029f-436c-89c6-093395c89235-tmp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:16.849219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.849197 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4"] Nov 01 21:45:16.849328 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.849312 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.881142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.881118 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 21:45:16.885247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.885229 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 21:45:16.915788 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.915769 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8qmmh\"" Nov 01 21:45:16.915915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.915832 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 21:45:16.915915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.915883 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 21:45:16.915915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.915888 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 21:45:16.915915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.915908 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 21:45:16.945451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945428 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945470 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-federate-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945499 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945633 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945727 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945674 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945789 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945773 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4kwv\" (UniqueName: \"kubernetes.io/projected/d8428012-b58b-4b52-ac62-85cce3271c5d-kube-api-access-p4kwv\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945817 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-serving-certs-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.945892 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.945852 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-metrics-client-ca\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:16.971143 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:16.971096 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bj7ms_bb80e501-2a27-4977-ae29-3804022b6d13/serve-healthcheck-canary/0.log" Nov 01 21:45:17.046347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046323 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-federate-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046352 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046546 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046606 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046555 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046622 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4kwv\" (UniqueName: \"kubernetes.io/projected/d8428012-b58b-4b52-ac62-85cce3271c5d-kube-api-access-p4kwv\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046709 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046664 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-serving-certs-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046709 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046688 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-metrics-client-ca\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.046857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.046718 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.047940 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.047914 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.048724 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.048703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-serving-certs-ca-bundle\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.049675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.049181 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8428012-b58b-4b52-ac62-85cce3271c5d-metrics-client-ca\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.049675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.049624 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-federate-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.049827 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.049780 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-telemeter-client-tls\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.050210 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.050189 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.051006 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.050985 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d8428012-b58b-4b52-ac62-85cce3271c5d-secret-telemeter-client\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.060278 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.060253 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4kwv\" (UniqueName: \"kubernetes.io/projected/d8428012-b58b-4b52-ac62-85cce3271c5d-kube-api-access-p4kwv\") pod \"telemeter-client-67c64ccbbb-m8ln4\" (UID: \"d8428012-b58b-4b52-ac62-85cce3271c5d\") " pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.205334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.205312 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" Nov 01 21:45:17.460945 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.460320 4395 generic.go:356] "Generic (PLEG): container finished" podID="5e367fa1-029f-436c-89c6-093395c89235" containerID="73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329" exitCode=0 Nov 01 21:45:17.460945 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.460382 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686fb85f67-vvzm9" event={"ID":"5e367fa1-029f-436c-89c6-093395c89235","Type":"ContainerDied","Data":"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329"} Nov 01 21:45:17.460945 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.460429 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686fb85f67-vvzm9" event={"ID":"5e367fa1-029f-436c-89c6-093395c89235","Type":"ContainerDied","Data":"dad1d2fa470b98cd8de575f21c73c1c88b407c53516be94e688f81c2b6bf7599"} Nov 01 21:45:17.460945 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.460456 4395 scope.go:122] "RemoveContainer" containerID="73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329" Nov 01 21:45:17.460945 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.460574 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686fb85f67-vvzm9" Nov 01 21:45:17.470980 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.470959 4395 scope.go:122] "RemoveContainer" containerID="73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329" Nov 01 21:45:17.471309 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:45:17.471281 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329\": container with ID starting with 73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329 not found: ID does not exist" containerID="73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329" Nov 01 21:45:17.471366 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.471322 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329"} err="failed to get container status \"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329\": rpc error: code = NotFound desc = could not find container \"73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329\": container with ID starting with 73e58e2562485bc8438446561e9ec7ceca6a47a6c09b5f8da43d6dc60fb54329 not found: ID does not exist" Nov 01 21:45:17.477319 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.477294 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:45:17.483789 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.483765 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-686fb85f67-vvzm9"] Nov 01 21:45:17.524290 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:17.524265 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4"] Nov 01 21:45:17.530489 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:45:17.530460 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8428012_b58b_4b52_ac62_85cce3271c5d.slice/crio-994015349cda17a9987e979e77f82dc784834fe4f2e72c1741d2848f677e29bd WatchSource:0}: Error finding container 994015349cda17a9987e979e77f82dc784834fe4f2e72c1741d2848f677e29bd: Status 404 returned error can't find the container with id 994015349cda17a9987e979e77f82dc784834fe4f2e72c1741d2848f677e29bd Nov 01 21:45:18.466629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:18.466593 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" event={"ID":"d8428012-b58b-4b52-ac62-85cce3271c5d","Type":"ContainerStarted","Data":"994015349cda17a9987e979e77f82dc784834fe4f2e72c1741d2848f677e29bd"} Nov 01 21:45:19.162639 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:19.162607 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e367fa1-029f-436c-89c6-093395c89235" path="/var/lib/kubelet/pods/5e367fa1-029f-436c-89c6-093395c89235/volumes" Nov 01 21:45:20.475384 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:20.475358 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" event={"ID":"d8428012-b58b-4b52-ac62-85cce3271c5d","Type":"ContainerStarted","Data":"87119f6b47c00ae9d02961d44adcb63f28322536ebef70663f8b432338c30a57"} Nov 01 21:45:22.299293 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:45:22.299263 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:45:22.484559 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.484523 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" event={"ID":"d8428012-b58b-4b52-ac62-85cce3271c5d","Type":"ContainerStarted","Data":"02856b445c8918f04805f530d3a52d2f7d5b7ad9dce8ab0c4816e39b9f70c9e0"} Nov 01 21:45:22.484695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.484590 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" event={"ID":"d8428012-b58b-4b52-ac62-85cce3271c5d","Type":"ContainerStarted","Data":"d028bea22d02e52f465ad54a2733769f1d7a05ef9d9e288d3dd460b2de15b83a"} Nov 01 21:45:22.511883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.511845 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67c64ccbbb-m8ln4" podStartSLOduration=2.234004536 podStartE2EDuration="6.51183007s" totalImagesPullingTime="4.277825534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:45:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:45:22.509574639 +0000 UTC m=+69.483477179" watchObservedRunningTime="2026-11-01 21:45:22.51183007 +0000 UTC m=+69.485732609" Nov 01 21:45:22.600983 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.600923 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:45:22.619429 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.619406 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.619617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.619420 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:45:22.687134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687103 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687141 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687166 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687323 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687281 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687323 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687316 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687397 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687377 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687397 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687393 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.687489 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.687456 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggk4\" (UniqueName: \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788404 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788383 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788413 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788436 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788463 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788738 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788542 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788738 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788568 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.788738 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.788600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggk4\" (UniqueName: \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791215 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791169 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791252 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791391 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791369 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791448 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791448 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791417 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.791553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.791449 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.793156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.793134 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.800396 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.800377 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggk4\" (UniqueName: \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\") pod \"console-5cd76f6bd9-xsnlq\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:22.944222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:22.944151 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:23.243459 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:23.243387 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:45:23.248420 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:45:23.248391 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730b11a9_d14c_4d57_8dcc_539b0a2772a4.slice/crio-15cc71c4288ab1cb337c420c690351664350120e40409b111cb0120bf4be0c36 WatchSource:0}: Error finding container 15cc71c4288ab1cb337c420c690351664350120e40409b111cb0120bf4be0c36: Status 404 returned error can't find the container with id 15cc71c4288ab1cb337c420c690351664350120e40409b111cb0120bf4be0c36 Nov 01 21:45:23.489348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:23.489312 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd76f6bd9-xsnlq" event={"ID":"730b11a9-d14c-4d57-8dcc-539b0a2772a4","Type":"ContainerStarted","Data":"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041"} Nov 01 21:45:23.489726 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:23.489358 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd76f6bd9-xsnlq" event={"ID":"730b11a9-d14c-4d57-8dcc-539b0a2772a4","Type":"ContainerStarted","Data":"15cc71c4288ab1cb337c420c690351664350120e40409b111cb0120bf4be0c36"} Nov 01 21:45:23.510412 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:23.510368 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cd76f6bd9-xsnlq" podStartSLOduration=1.510354505 podStartE2EDuration="1.510354505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:45:23.509532278 +0000 UTC m=+70.483434816" watchObservedRunningTime="2026-11-01 21:45:23.510354505 +0000 UTC m=+70.484257043" Nov 01 21:45:32.948060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:32.947783 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:32.948793 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:32.948547 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:32.952843 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:32.952822 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:33.524781 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:33.524757 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:45:33.575853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:33.575822 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:45:52.304529 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:45:52.304486 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:45:58.597756 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.597689 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6555c7dbc8-8fzbt" podUID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" containerName="console" containerID="cri-o://fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c" gracePeriod=15 Nov 01 21:45:58.839904 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.839881 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:45:58.935603 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935546 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\" (UniqueName: \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935603 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935587 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935635 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935664 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935739 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935917 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935780 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935917 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935815 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935917 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935853 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\") pod \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\" (UID: \"acd89f58-f3b8-447e-99a5-0c0f2ff479aa\") " Nov 01 21:45:58.935917 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.935899 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:45:58.936159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.936140 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-tmp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:58.936220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.936166 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:58.936220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.936203 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:58.936314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.936268 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:58.936765 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.936678 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:45:58.937870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.937849 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "kube-api-access-zrvpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:45:58.937967 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.937940 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:58.938117 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:58.938103 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert" pod "acd89f58-f3b8-447e-99a5-0c0f2ff479aa" (UID: "acd89f58-f3b8-447e-99a5-0c0f2ff479aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:45:59.036758 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036735 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036758 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036756 4395 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-trusted-ca-bundle\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036765 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-oauth-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036775 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-console-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036783 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvpp\" (UniqueName: \"kubernetes.io/projected/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-kube-api-access-zrvpp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036793 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.036872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.036801 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd89f58-f3b8-447e-99a5-0c0f2ff479aa-oauth-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:45:59.616121 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.616085 4395 generic.go:356] "Generic (PLEG): container finished" podID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" containerID="fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c" exitCode=0 Nov 01 21:45:59.616548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.616136 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6555c7dbc8-8fzbt" Nov 01 21:45:59.616548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.616181 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6555c7dbc8-8fzbt" event={"ID":"acd89f58-f3b8-447e-99a5-0c0f2ff479aa","Type":"ContainerDied","Data":"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c"} Nov 01 21:45:59.616548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.616238 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6555c7dbc8-8fzbt" event={"ID":"acd89f58-f3b8-447e-99a5-0c0f2ff479aa","Type":"ContainerDied","Data":"b8a4062306dbc697646f836c134972ce2924e9ea8b1ef4c163b681f414751631"} Nov 01 21:45:59.616548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.616258 4395 scope.go:122] "RemoveContainer" containerID="fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c" Nov 01 21:45:59.624531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.624513 4395 scope.go:122] "RemoveContainer" containerID="fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c" Nov 01 21:45:59.624860 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:45:59.624840 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c\": container with ID starting with fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c not found: ID does not exist" containerID="fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c" Nov 01 21:45:59.624906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.624869 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c"} err="failed to get container status \"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c\": rpc error: code = NotFound desc = could not find container \"fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c\": container with ID starting with fa419a47cac2bbb1c2a88406d5ccb3e2f6415f27c2042af5b99a67618599eb5c not found: ID does not exist" Nov 01 21:45:59.634262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.634240 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:45:59.638241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:45:59.638219 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6555c7dbc8-8fzbt"] Nov 01 21:46:01.162488 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:01.162454 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" path="/var/lib/kubelet/pods/acd89f58-f3b8-447e-99a5-0c0f2ff479aa/volumes" Nov 01 21:46:07.963610 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.963572 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7486c6f8cc-7jrh4"] Nov 01 21:46:07.964112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.963848 4395 cpu_manager.go:414] "containerMap: removing container" podUID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" containerName="console" Nov 01 21:46:07.964112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.963864 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" containerName="console" Nov 01 21:46:07.964112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.963934 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd89f58-f3b8-447e-99a5-0c0f2ff479aa" containerName="console" Nov 01 21:46:07.968124 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.968106 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:07.978531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:07.978508 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7486c6f8cc-7jrh4"] Nov 01 21:46:08.110061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110013 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-tmp\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110076 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110172 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110261 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110197 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-service-ca\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110261 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110246 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-oauth-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110321 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110287 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-oauth-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110321 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110310 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmf2\" (UniqueName: \"kubernetes.io/projected/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-kube-api-access-8nmf2\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.110382 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.110326 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-trusted-ca-bundle\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211601 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211569 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-tmp\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211601 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211604 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211620 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211636 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-service-ca\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211653 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-oauth-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211682 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-oauth-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmf2\" (UniqueName: \"kubernetes.io/projected/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-kube-api-access-8nmf2\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.211754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.211721 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-trusted-ca-bundle\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.212073 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.212050 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-tmp\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.212405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.212383 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-oauth-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.212405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.212396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-service-ca\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.212678 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.212655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-trusted-ca-bundle\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.212864 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.212843 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.214420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.214384 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-serving-cert\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.214420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.214411 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-console-oauth-config\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.224056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.224021 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmf2\" (UniqueName: \"kubernetes.io/projected/c2298c97-73a2-4ef7-8cea-8f2aad5190c8-kube-api-access-8nmf2\") pod \"console-7486c6f8cc-7jrh4\" (UID: \"c2298c97-73a2-4ef7-8cea-8f2aad5190c8\") " pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.279598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.279580 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:08.553566 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.553537 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7486c6f8cc-7jrh4"] Nov 01 21:46:08.556290 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:46:08.556265 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2298c97_73a2_4ef7_8cea_8f2aad5190c8.slice/crio-a18b1c5c7c01957fd6dcd3e0e77961de447a1785005b5589b9ee23bf113abf52 WatchSource:0}: Error finding container a18b1c5c7c01957fd6dcd3e0e77961de447a1785005b5589b9ee23bf113abf52: Status 404 returned error can't find the container with id a18b1c5c7c01957fd6dcd3e0e77961de447a1785005b5589b9ee23bf113abf52 Nov 01 21:46:08.651123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.651095 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486c6f8cc-7jrh4" event={"ID":"c2298c97-73a2-4ef7-8cea-8f2aad5190c8","Type":"ContainerStarted","Data":"05bf6b41e1ab3953bd073d0e3675ef76128a1c651179a50bc96b8f8e4b990b20"} Nov 01 21:46:08.651267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.651148 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486c6f8cc-7jrh4" event={"ID":"c2298c97-73a2-4ef7-8cea-8f2aad5190c8","Type":"ContainerStarted","Data":"a18b1c5c7c01957fd6dcd3e0e77961de447a1785005b5589b9ee23bf113abf52"} Nov 01 21:46:08.672217 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:08.672174 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7486c6f8cc-7jrh4" podStartSLOduration=1.672159537 podStartE2EDuration="1.672159537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:46:08.671515514 +0000 UTC m=+115.645418057" watchObservedRunningTime="2026-11-01 21:46:08.672159537 +0000 UTC m=+115.646062076" Nov 01 21:46:18.280138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:18.280107 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:18.280529 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:18.280168 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:18.284604 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:18.284582 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:18.689300 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:18.689236 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7486c6f8cc-7jrh4" Nov 01 21:46:18.736606 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:18.736573 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:46:22.300332 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:46:22.300287 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:46:43.756100 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:43.756000 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5cd76f6bd9-xsnlq" podUID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" containerName="console" containerID="cri-o://3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041" gracePeriod=15 Nov 01 21:46:43.983340 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:43.983316 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:46:44.055681 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055606 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055698 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055773 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055803 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055818 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:46:44.055953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055827 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055852 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055873 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.055953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.055896 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\" (UniqueName: \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\") pod \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\" (UID: \"730b11a9-d14c-4d57-8dcc-539b0a2772a4\") " Nov 01 21:46:44.056222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.056114 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730b11a9-d14c-4d57-8dcc-539b0a2772a4-tmp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.056222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.056113 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:44.056332 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.056249 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:44.056386 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.056356 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:44.056386 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.056369 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:46:44.058225 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.058206 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "kube-api-access-fggk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:46:44.058510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.058493 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:46:44.058567 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.058525 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config" pod "730b11a9-d14c-4d57-8dcc-539b0a2772a4" (UID: "730b11a9-d14c-4d57-8dcc-539b0a2772a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:46:44.157359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157322 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157352 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-oauth-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157365 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157378 4395 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-trusted-ca-bundle\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157390 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-oauth-serving-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157401 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggk4\" (UniqueName: \"kubernetes.io/projected/730b11a9-d14c-4d57-8dcc-539b0a2772a4-kube-api-access-fggk4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.157591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.157414 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/730b11a9-d14c-4d57-8dcc-539b0a2772a4-console-config\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:46:44.767486 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.767443 4395 generic.go:356] "Generic (PLEG): container finished" podID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" containerID="3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041" exitCode=0 Nov 01 21:46:44.767866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.767516 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd76f6bd9-xsnlq" Nov 01 21:46:44.767866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.767517 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd76f6bd9-xsnlq" event={"ID":"730b11a9-d14c-4d57-8dcc-539b0a2772a4","Type":"ContainerDied","Data":"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041"} Nov 01 21:46:44.767866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.767666 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd76f6bd9-xsnlq" event={"ID":"730b11a9-d14c-4d57-8dcc-539b0a2772a4","Type":"ContainerDied","Data":"15cc71c4288ab1cb337c420c690351664350120e40409b111cb0120bf4be0c36"} Nov 01 21:46:44.767866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.767694 4395 scope.go:122] "RemoveContainer" containerID="3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041" Nov 01 21:46:44.775857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.775837 4395 scope.go:122] "RemoveContainer" containerID="3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041" Nov 01 21:46:44.776167 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:46:44.776149 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041\": container with ID starting with 3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041 not found: ID does not exist" containerID="3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041" Nov 01 21:46:44.776210 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.776176 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041"} err="failed to get container status \"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041\": rpc error: code = NotFound desc = could not find container \"3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041\": container with ID starting with 3b21a2976a64b318e9eabb5f69a56649b67c3737c8fff369c62d72a2bc81b041 not found: ID does not exist" Nov 01 21:46:44.788971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.788942 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:46:44.792871 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:44.792852 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd76f6bd9-xsnlq"] Nov 01 21:46:45.161912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:46:45.161839 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" path="/var/lib/kubelet/pods/730b11a9-d14c-4d57-8dcc-539b0a2772a4/volumes" Nov 01 21:46:52.299009 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:46:52.298973 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:47:22.302071 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:47:22.302018 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:47:52.299645 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:47:52.299601 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:48:00.552371 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.552280 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n"] Nov 01 21:48:00.552847 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.552558 4395 cpu_manager.go:414] "containerMap: removing container" podUID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" containerName="console" Nov 01 21:48:00.552847 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.552569 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" containerName="console" Nov 01 21:48:00.552847 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.552621 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="730b11a9-d14c-4d57-8dcc-539b0a2772a4" containerName="console" Nov 01 21:48:00.555623 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.555606 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.562834 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.562815 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 21:48:00.562990 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.562970 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nmgwg\"" Nov 01 21:48:00.564274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.564246 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 21:48:00.565588 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.565567 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n"] Nov 01 21:48:00.621707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.621681 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk"] Nov 01 21:48:00.624793 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.624778 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.639436 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.639411 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk"] Nov 01 21:48:00.661600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.661578 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qkp5d\"" Nov 01 21:48:00.686873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.686853 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.686985 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.686908 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.686985 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.686929 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tch4\" (UniqueName: \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.788167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788146 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.788238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788177 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.788238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788199 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tch4\" (UniqueName: \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.788238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788220 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.788379 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788283 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.788379 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788301 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w44jd\" (UniqueName: \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.788553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788533 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.788618 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.788586 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.801133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.801115 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tch4\" (UniqueName: \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.872082 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.872018 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:00.889067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.889040 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.889159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.889077 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w44jd\" (UniqueName: \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.889159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.889104 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.889405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.889390 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.889490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.889470 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.902466 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.902445 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w44jd\" (UniqueName: \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:00.939709 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:00.939683 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:01.058757 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.058730 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:48:01.062019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.062005 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.065879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065834 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 21:48:01.065879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065844 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 21:48:01.065879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065844 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 21:48:01.066155 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065881 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 21:48:01.066155 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065960 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 21:48:01.066155 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.065963 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 21:48:01.066155 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.066048 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8pxdd\"" Nov 01 21:48:01.077380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.077359 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:48:01.147504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.147487 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n"] Nov 01 21:48:01.149902 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:48:01.149878 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40b736a_4037_4def_af9b_140e57cee94b.slice/crio-c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7 WatchSource:0}: Error finding container c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7: Status 404 returned error can't find the container with id c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7 Nov 01 21:48:01.192101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192077 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192194 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192255 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192221 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvxh\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192283 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192383 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192365 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192399 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.192440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.192432 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.214569 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.214551 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk"] Nov 01 21:48:01.217482 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:48:01.217461 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ec41d3_c66c_41c8_a627_adbede982666.slice/crio-39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff WatchSource:0}: Error finding container 39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff: Status 404 returned error can't find the container with id 39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff Nov 01 21:48:01.293092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293069 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293103 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvxh\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293131 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293156 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293173 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293377 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293192 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293377 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293209 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.293966 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.293769 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.295487 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.295467 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.295672 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.295653 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.295752 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.295732 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.295792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.295735 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.305293 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.305273 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.305374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.305277 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvxh\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\") pod \"istiod-openshift-gateway-7876bfcb74-zpvwt\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.372860 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.372809 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:01.648125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:01.648059 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:48:01.650471 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:48:01.650442 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8a0748_c612_44d3_ad7b_303f948ce09b.slice/crio-66fe64e3376ee2e0121c2124941042128c8b075fa1ab177d547ea9dc8a22cca1 WatchSource:0}: Error finding container 66fe64e3376ee2e0121c2124941042128c8b075fa1ab177d547ea9dc8a22cca1: Status 404 returned error can't find the container with id 66fe64e3376ee2e0121c2124941042128c8b075fa1ab177d547ea9dc8a22cca1 Nov 01 21:48:02.023412 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:02.023326 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" event={"ID":"c40b736a-4037-4def-af9b-140e57cee94b","Type":"ContainerStarted","Data":"c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7"} Nov 01 21:48:02.025330 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:02.025258 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" event={"ID":"28ec41d3-c66c-41c8-a627-adbede982666","Type":"ContainerStarted","Data":"39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff"} Nov 01 21:48:02.027841 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:02.027784 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" event={"ID":"7e8a0748-c612-44d3-ad7b-303f948ce09b","Type":"ContainerStarted","Data":"66fe64e3376ee2e0121c2124941042128c8b075fa1ab177d547ea9dc8a22cca1"} Nov 01 21:48:05.316633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:05.316597 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:48:05.316903 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:05.316682 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:48:06.050502 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:06.050458 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" event={"ID":"7e8a0748-c612-44d3-ad7b-303f948ce09b","Type":"ContainerStarted","Data":"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3"} Nov 01 21:48:06.051444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:06.051419 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:06.053285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:06.053263 4395 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zpvwt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:48:06.053405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:06.053303 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:48:06.078472 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:06.078421 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" podStartSLOduration=1.415123038 podStartE2EDuration="5.078406509s" totalImagesPullingTime="3.663283471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:06.076570341 +0000 UTC m=+233.050472878" watchObservedRunningTime="2026-11-01 21:48:06.078406509 +0000 UTC m=+233.052309047" Nov 01 21:48:07.055515 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:07.055479 4395 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zpvwt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:48:07.056080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:07.055552 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:48:08.059815 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:08.059783 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:48:09.068759 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:09.068728 4395 generic.go:356] "Generic (PLEG): container finished" podID="c40b736a-4037-4def-af9b-140e57cee94b" containerID="3849b91be444a9361aad2b459813247ced4b521b179d5da9227611d889b7bb23" exitCode=0 Nov 01 21:48:09.069233 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:09.068815 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" event={"ID":"c40b736a-4037-4def-af9b-140e57cee94b","Type":"ContainerDied","Data":"3849b91be444a9361aad2b459813247ced4b521b179d5da9227611d889b7bb23"} Nov 01 21:48:09.070532 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:09.070503 4395 generic.go:356] "Generic (PLEG): container finished" podID="28ec41d3-c66c-41c8-a627-adbede982666" containerID="8d677ba9656a5f06bfbdf145ff26d4e2c65184230cc2cf31f501c2d034405d88" exitCode=0 Nov 01 21:48:09.070639 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:09.070548 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" event={"ID":"28ec41d3-c66c-41c8-a627-adbede982666","Type":"ContainerDied","Data":"8d677ba9656a5f06bfbdf145ff26d4e2c65184230cc2cf31f501c2d034405d88"} Nov 01 21:48:12.084679 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:12.084647 4395 generic.go:356] "Generic (PLEG): container finished" podID="c40b736a-4037-4def-af9b-140e57cee94b" containerID="fe14923d84c3ab0f0c3f9cdb1e0a764bf18f2648b656c724760b01a629a3436c" exitCode=0 Nov 01 21:48:12.085089 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:12.084738 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" event={"ID":"c40b736a-4037-4def-af9b-140e57cee94b","Type":"ContainerDied","Data":"fe14923d84c3ab0f0c3f9cdb1e0a764bf18f2648b656c724760b01a629a3436c"} Nov 01 21:48:15.098971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:15.098938 4395 generic.go:356] "Generic (PLEG): container finished" podID="28ec41d3-c66c-41c8-a627-adbede982666" containerID="e7db931b5e34931d8660e2267f585d6be39392f0ac99ed58f4908c5f357a6d85" exitCode=0 Nov 01 21:48:15.099450 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:15.098980 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" event={"ID":"28ec41d3-c66c-41c8-a627-adbede982666","Type":"ContainerDied","Data":"e7db931b5e34931d8660e2267f585d6be39392f0ac99ed58f4908c5f357a6d85"} Nov 01 21:48:19.119488 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:19.119456 4395 generic.go:356] "Generic (PLEG): container finished" podID="c40b736a-4037-4def-af9b-140e57cee94b" containerID="888c64a297e0c68c43f999174f5afe1e843f4bf3656bf646c433a51aa2d8fa0f" exitCode=0 Nov 01 21:48:19.119860 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:19.119519 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" event={"ID":"c40b736a-4037-4def-af9b-140e57cee94b","Type":"ContainerDied","Data":"888c64a297e0c68c43f999174f5afe1e843f4bf3656bf646c433a51aa2d8fa0f"} Nov 01 21:48:20.255089 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.255065 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:20.351214 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.351184 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\") pod \"c40b736a-4037-4def-af9b-140e57cee94b\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " Nov 01 21:48:20.351381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.351255 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\") pod \"c40b736a-4037-4def-af9b-140e57cee94b\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " Nov 01 21:48:20.351381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.351309 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\" (UniqueName: \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\") pod \"c40b736a-4037-4def-af9b-140e57cee94b\" (UID: \"c40b736a-4037-4def-af9b-140e57cee94b\") " Nov 01 21:48:20.351851 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.351825 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle" pod "c40b736a-4037-4def-af9b-140e57cee94b" (UID: "c40b736a-4037-4def-af9b-140e57cee94b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:20.353872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.353844 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4" pod "c40b736a-4037-4def-af9b-140e57cee94b" (UID: "c40b736a-4037-4def-af9b-140e57cee94b"). InnerVolumeSpecName "kube-api-access-5tch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:20.357845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.357820 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util" pod "c40b736a-4037-4def-af9b-140e57cee94b" (UID: "c40b736a-4037-4def-af9b-140e57cee94b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:20.452438 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.452389 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-bundle\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:20.452438 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.452410 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c40b736a-4037-4def-af9b-140e57cee94b-util\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:20.452438 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:20.452418 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tch4\" (UniqueName: \"kubernetes.io/projected/c40b736a-4037-4def-af9b-140e57cee94b-kube-api-access-5tch4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:21.128244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:21.128219 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" Nov 01 21:48:21.128424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:21.128215 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wt64n" event={"ID":"c40b736a-4037-4def-af9b-140e57cee94b","Type":"ContainerDied","Data":"c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7"} Nov 01 21:48:21.128424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:21.128346 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45df1a21cf1cf0a786b97d1313da22ac7d4ec24c6c9651e417a3cc203c31fb7" Nov 01 21:48:22.133748 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:22.133709 4395 generic.go:356] "Generic (PLEG): container finished" podID="28ec41d3-c66c-41c8-a627-adbede982666" containerID="c38ebe63a4d2b24d4f6eb9a441dfe94d50295a760e05b538c2ea273735c95aaa" exitCode=0 Nov 01 21:48:22.134163 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:22.133794 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" event={"ID":"28ec41d3-c66c-41c8-a627-adbede982666","Type":"ContainerDied","Data":"c38ebe63a4d2b24d4f6eb9a441dfe94d50295a760e05b538c2ea273735c95aaa"} Nov 01 21:48:22.300728 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:48:22.300682 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:48:23.329188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.329157 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:23.372987 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.372951 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\") pod \"28ec41d3-c66c-41c8-a627-adbede982666\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " Nov 01 21:48:23.372987 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.372990 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\") pod \"28ec41d3-c66c-41c8-a627-adbede982666\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " Nov 01 21:48:23.373208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.373017 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\" (UniqueName: \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\") pod \"28ec41d3-c66c-41c8-a627-adbede982666\" (UID: \"28ec41d3-c66c-41c8-a627-adbede982666\") " Nov 01 21:48:23.373721 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.373696 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle" pod "28ec41d3-c66c-41c8-a627-adbede982666" (UID: "28ec41d3-c66c-41c8-a627-adbede982666"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:23.375279 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.375254 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd" pod "28ec41d3-c66c-41c8-a627-adbede982666" (UID: "28ec41d3-c66c-41c8-a627-adbede982666"). InnerVolumeSpecName "kube-api-access-w44jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:48:23.378605 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.378581 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util" pod "28ec41d3-c66c-41c8-a627-adbede982666" (UID: "28ec41d3-c66c-41c8-a627-adbede982666"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:48:23.474415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.474349 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-bundle\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:23.474415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.474372 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28ec41d3-c66c-41c8-a627-adbede982666-util\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:23.474415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:23.474383 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w44jd\" (UniqueName: \"kubernetes.io/projected/28ec41d3-c66c-41c8-a627-adbede982666-kube-api-access-w44jd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:48:24.142713 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:24.142677 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" event={"ID":"28ec41d3-c66c-41c8-a627-adbede982666","Type":"ContainerDied","Data":"39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff"} Nov 01 21:48:24.142904 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:24.142734 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ced4ea58d1901cb907a976bf039d27100cab5f052fcb2970dc08b5b3ea46ff" Nov 01 21:48:24.142904 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:24.142695 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c59rgk" Nov 01 21:48:39.773741 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773705 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k"] Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773948 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="pull" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773958 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="pull" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773973 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="util" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773979 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="util" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773986 4395 cpu_manager.go:414] "containerMap: removing container" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="extract" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.773991 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="extract" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774002 4395 cpu_manager.go:414] "containerMap: removing container" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="pull" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774006 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="pull" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774018 4395 cpu_manager.go:414] "containerMap: removing container" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="util" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774022 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="util" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774044 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="extract" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774048 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="extract" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774086 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40b736a-4037-4def-af9b-140e57cee94b" containerName="extract" Nov 01 21:48:39.775888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.774094 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ec41d3-c66c-41c8-a627-adbede982666" containerName="extract" Nov 01 21:48:39.776610 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.776596 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.810278 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.810249 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k"] Nov 01 21:48:39.868109 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868088 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 21:48:39.868247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868096 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 21:48:39.868247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868139 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 21:48:39.868247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868147 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 21:48:39.868247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868225 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 21:48:39.868398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.868280 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wbcjm\"" Nov 01 21:48:39.885554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885533 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.885636 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885569 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.885636 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885588 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.885711 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885682 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.885767 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885752 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4k7f\" (UniqueName: \"kubernetes.io/projected/6bd474d8-3c94-4873-929a-f0caa9eba09a-kube-api-access-z4k7f\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.885803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.885776 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.986899 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.986874 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.986989 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.986903 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.986989 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.986927 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.986989 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.986974 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4k7f\" (UniqueName: \"kubernetes.io/projected/6bd474d8-3c94-4873-929a-f0caa9eba09a-kube-api-access-z4k7f\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.987134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.986998 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.987134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.987060 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.987303 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.987282 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.989477 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.989450 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bd474d8-3c94-4873-929a-f0caa9eba09a-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.989579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.989563 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.989579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.989566 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:39.989741 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:39.989721 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd474d8-3c94-4873-929a-f0caa9eba09a-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:40.005909 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:40.005886 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4k7f\" (UniqueName: \"kubernetes.io/projected/6bd474d8-3c94-4873-929a-f0caa9eba09a-kube-api-access-z4k7f\") pod \"opendatahub-operator-controller-manager-98c48f84c-ncl5k\" (UID: \"6bd474d8-3c94-4873-929a-f0caa9eba09a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:40.090234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:40.090183 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:40.390612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:40.390589 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k"] Nov 01 21:48:40.393127 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:48:40.393105 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd474d8_3c94_4873_929a_f0caa9eba09a.slice/crio-a6c97a77fd1d036bd9bf414e6d85225baada701470fcc077525f52367f23f324 WatchSource:0}: Error finding container a6c97a77fd1d036bd9bf414e6d85225baada701470fcc077525f52367f23f324: Status 404 returned error can't find the container with id a6c97a77fd1d036bd9bf414e6d85225baada701470fcc077525f52367f23f324 Nov 01 21:48:41.220738 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:41.220696 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" event={"ID":"6bd474d8-3c94-4873-929a-f0caa9eba09a","Type":"ContainerStarted","Data":"a6c97a77fd1d036bd9bf414e6d85225baada701470fcc077525f52367f23f324"} Nov 01 21:48:44.233074 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:44.232962 4395 generic.go:356] "Generic (PLEG): container finished" podID="6bd474d8-3c94-4873-929a-f0caa9eba09a" containerID="494cdb5873fc7fb94522d6a8905f1dce9d780055b58d4cf498e228cc63dc2f6a" exitCode=0 Nov 01 21:48:44.233074 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:44.233009 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" event={"ID":"6bd474d8-3c94-4873-929a-f0caa9eba09a","Type":"ContainerDied","Data":"494cdb5873fc7fb94522d6a8905f1dce9d780055b58d4cf498e228cc63dc2f6a"} Nov 01 21:48:45.238436 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:45.238401 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" event={"ID":"6bd474d8-3c94-4873-929a-f0caa9eba09a","Type":"ContainerStarted","Data":"c21fa9fa69e0a79250764d7283b800395f5d5d42856cd8ae8798ea728b4df647"} Nov 01 21:48:45.238928 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:45.238509 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:48:45.288600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:45.288540 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" podStartSLOduration=2.316706439 podStartE2EDuration="6.288524081s" totalImagesPullingTime="3.513638948s" totalInitContainerRuntime="458.178694ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:45.287282557 +0000 UTC m=+272.261185098" watchObservedRunningTime="2026-11-01 21:48:45.288524081 +0000 UTC m=+272.262426619" Nov 01 21:48:52.299401 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:48:52.299364 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:48:56.244342 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:48:56.244309 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-ncl5k" Nov 01 21:49:12.151390 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.151354 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4"] Nov 01 21:49:12.154714 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.154695 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.168802 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.168773 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4"] Nov 01 21:49:12.238201 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.238166 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 21:49:12.238369 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.238304 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 21:49:12.238369 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.238316 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8nmd\"" Nov 01 21:49:12.249671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.249647 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.249671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.249686 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tmp\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.249925 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.249857 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dktl\" (UniqueName: \"kubernetes.io/projected/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-kube-api-access-5dktl\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.350727 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.350700 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.350727 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.350730 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tmp\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.350962 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.350758 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dktl\" (UniqueName: \"kubernetes.io/projected/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-kube-api-access-5dktl\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.350962 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:49:12.350878 4395 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Nov 01 21:49:12.350962 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:49:12.350953 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs podName:aca37cfc-3c06-4afc-a12d-20cd8600f5a9 nodeName:}" failed. No retries permitted until 2026-11-01 21:49:12.850935904 +0000 UTC m=+299.824838434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs") pod "kube-auth-proxy-b69dc6f4d-9rkd4" (UID: "aca37cfc-3c06-4afc-a12d-20cd8600f5a9") : secret "kube-auth-proxy-tls" not found Nov 01 21:49:12.353214 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.353188 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tmp\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.369637 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.369604 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dktl\" (UniqueName: \"kubernetes.io/projected/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-kube-api-access-5dktl\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.855003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.854965 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:12.857589 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:12.857563 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca37cfc-3c06-4afc-a12d-20cd8600f5a9-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-9rkd4\" (UID: \"aca37cfc-3c06-4afc-a12d-20cd8600f5a9\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:13.066780 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:13.066738 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" Nov 01 21:49:13.351554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:13.351522 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4"] Nov 01 21:49:13.355417 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:49:13.355383 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca37cfc_3c06_4afc_a12d_20cd8600f5a9.slice/crio-1b27af00303ab51a57e322e3790dab7991ed6a9060da8c1eef119e816afdaa2b WatchSource:0}: Error finding container 1b27af00303ab51a57e322e3790dab7991ed6a9060da8c1eef119e816afdaa2b: Status 404 returned error can't find the container with id 1b27af00303ab51a57e322e3790dab7991ed6a9060da8c1eef119e816afdaa2b Nov 01 21:49:14.351624 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.351581 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" event={"ID":"aca37cfc-3c06-4afc-a12d-20cd8600f5a9","Type":"ContainerStarted","Data":"1b27af00303ab51a57e322e3790dab7991ed6a9060da8c1eef119e816afdaa2b"} Nov 01 21:49:14.391867 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.391832 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4"] Nov 01 21:49:14.395437 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.395411 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.410781 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.410750 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 21:49:14.410948 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.410890 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 21:49:14.411011 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.410958 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 21:49:14.411772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.411748 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4"] Nov 01 21:49:14.441128 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.441092 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 21:49:14.441292 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.441222 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6fqfs\"" Nov 01 21:49:14.572455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.572093 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b221c69-1f05-4a39-8186-66503ad4055c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.572455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.572228 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b221c69-1f05-4a39-8186-66503ad4055c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.572455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.572318 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpd5\" (UniqueName: \"kubernetes.io/projected/7b221c69-1f05-4a39-8186-66503ad4055c-kube-api-access-hxpd5\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.673410 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.673322 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b221c69-1f05-4a39-8186-66503ad4055c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.673410 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.673408 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b221c69-1f05-4a39-8186-66503ad4055c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.673629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.673476 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpd5\" (UniqueName: \"kubernetes.io/projected/7b221c69-1f05-4a39-8186-66503ad4055c-kube-api-access-hxpd5\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.674289 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.674261 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b221c69-1f05-4a39-8186-66503ad4055c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.676285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.676264 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b221c69-1f05-4a39-8186-66503ad4055c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.688534 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.688500 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpd5\" (UniqueName: \"kubernetes.io/projected/7b221c69-1f05-4a39-8186-66503ad4055c-kube-api-access-hxpd5\") pod \"kuadrant-console-plugin-66c4fb575d-4qfv4\" (UID: \"7b221c69-1f05-4a39-8186-66503ad4055c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.717513 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.717489 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" Nov 01 21:49:14.998657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:14.998629 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4"] Nov 01 21:49:15.000597 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:49:15.000561 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b221c69_1f05_4a39_8186_66503ad4055c.slice/crio-fa52d60e6fc9a259969e727739ca7eea11f1b19b6f757c059a763c16b8e0222a WatchSource:0}: Error finding container fa52d60e6fc9a259969e727739ca7eea11f1b19b6f757c059a763c16b8e0222a: Status 404 returned error can't find the container with id fa52d60e6fc9a259969e727739ca7eea11f1b19b6f757c059a763c16b8e0222a Nov 01 21:49:15.358503 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:15.358463 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" event={"ID":"7b221c69-1f05-4a39-8186-66503ad4055c","Type":"ContainerStarted","Data":"fa52d60e6fc9a259969e727739ca7eea11f1b19b6f757c059a763c16b8e0222a"} Nov 01 21:49:15.866608 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:15.866585 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 21:49:16.367365 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:16.367325 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" event={"ID":"aca37cfc-3c06-4afc-a12d-20cd8600f5a9","Type":"ContainerStarted","Data":"736832eee93f833b613e2b2c82a713299076e4d05191718aa9a94f62fbafe1a7"} Nov 01 21:49:16.390569 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:16.390512 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-9rkd4" podStartSLOduration=1.88519374 podStartE2EDuration="4.390494383s" totalImagesPullingTime="2.505300643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:16.39034418 +0000 UTC m=+303.364246719" watchObservedRunningTime="2026-11-01 21:49:16.390494383 +0000 UTC m=+303.364396922" Nov 01 21:49:21.391970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:21.391934 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" event={"ID":"7b221c69-1f05-4a39-8186-66503ad4055c","Type":"ContainerStarted","Data":"ebda040802c917336e2f638bd1d76be01dcb05eb054070987d609635361f7533"} Nov 01 21:49:21.409931 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:21.409876 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4qfv4" podStartSLOduration=1.604017195 podStartE2EDuration="7.409862131s" totalImagesPullingTime="5.805844936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:21.40848391 +0000 UTC m=+308.382386460" watchObservedRunningTime="2026-11-01 21:49:21.409862131 +0000 UTC m=+308.383764669" Nov 01 21:49:22.300363 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:49:22.300329 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:49:29.785979 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.785896 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d"] Nov 01 21:49:29.789171 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.789157 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:29.796934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.796905 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 21:49:29.797076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.796930 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 21:49:29.797845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.797823 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 21:49:29.797993 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.797931 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7zln7\"" Nov 01 21:49:29.798399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.798377 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d"] Nov 01 21:49:29.895815 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.895787 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12a3a283-9b34-410d-80f7-77b82731ff51-data\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:29.895949 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.895831 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlchk\" (UniqueName: \"kubernetes.io/projected/12a3a283-9b34-410d-80f7-77b82731ff51-kube-api-access-nlchk\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:29.996197 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.996168 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12a3a283-9b34-410d-80f7-77b82731ff51-data\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:29.996336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.996212 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlchk\" (UniqueName: \"kubernetes.io/projected/12a3a283-9b34-410d-80f7-77b82731ff51-kube-api-access-nlchk\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:29.996597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:29.996575 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12a3a283-9b34-410d-80f7-77b82731ff51-data\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:30.008288 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:30.008265 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlchk\" (UniqueName: \"kubernetes.io/projected/12a3a283-9b34-410d-80f7-77b82731ff51-kube-api-access-nlchk\") pod \"postgres-7cdbf6dc44-k248d\" (UID: \"12a3a283-9b34-410d-80f7-77b82731ff51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:30.101577 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:30.101520 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:30.378810 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:30.378730 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d"] Nov 01 21:49:30.382250 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:49:30.382213 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a3a283_9b34_410d_80f7_77b82731ff51.slice/crio-1704997cd09deee1d8e3b9a52ea5af76d62fad3943d9dd3fb0d38deb7a794235 WatchSource:0}: Error finding container 1704997cd09deee1d8e3b9a52ea5af76d62fad3943d9dd3fb0d38deb7a794235: Status 404 returned error can't find the container with id 1704997cd09deee1d8e3b9a52ea5af76d62fad3943d9dd3fb0d38deb7a794235 Nov 01 21:49:30.384123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:30.384104 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:49:30.427516 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:30.427491 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" event={"ID":"12a3a283-9b34-410d-80f7-77b82731ff51","Type":"ContainerStarted","Data":"1704997cd09deee1d8e3b9a52ea5af76d62fad3943d9dd3fb0d38deb7a794235"} Nov 01 21:49:36.453438 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:36.453399 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" event={"ID":"12a3a283-9b34-410d-80f7-77b82731ff51","Type":"ContainerStarted","Data":"4f29ebaf7392c3a8dd09e9aa6f429da518b930828d4cdcb2b69e0dfb9d093994"} Nov 01 21:49:36.453809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:36.453496 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:36.472208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:36.472161 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" podStartSLOduration=1.908604644 podStartE2EDuration="7.472145374s" totalImagesPullingTime="5.56354073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:36.471102156 +0000 UTC m=+323.445004696" watchObservedRunningTime="2026-11-01 21:49:36.472145374 +0000 UTC m=+323.446047915" Nov 01 21:49:42.486666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:42.486636 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k248d" Nov 01 21:49:49.404231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.404199 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:49:49.670622 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.670545 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:49:49.670767 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.670662 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.676119 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.675991 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ntfs9\"" Nov 01 21:49:49.676119 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.676074 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 21:49:49.676348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.676127 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 21:49:49.676348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.676175 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 21:49:49.676348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.676180 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 21:49:49.676348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.676219 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 21:49:49.722407 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.722382 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 21:49:49.751867 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.751837 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.752005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.751875 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.752005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.751899 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnr4c\" (UniqueName: \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.752515 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.752487 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 21:49:49.752615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.752575 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.853311 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853279 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nn2\" (UniqueName: \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.853472 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853321 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.853472 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853343 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.853598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853497 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.853598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853537 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.853598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.853580 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnr4c\" (UniqueName: \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.856136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.856115 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.856309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.856151 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.866021 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.866002 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnr4c\" (UniqueName: \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\") pod \"maas-controller-579cf7c6d-sk98t\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:49.953893 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.953849 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nn2\" (UniqueName: \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.953893 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.953879 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.953893 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.953895 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.956184 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.956162 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.956310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.956294 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:49.968233 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:49.968215 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nn2\" (UniqueName: \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\") pod \"maas-controller-56cb88789f-6nn87\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:50.012072 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.012048 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:50.066925 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.066904 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:50.318328 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.318301 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:49:50.321399 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:49:50.321370 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc842afcd_e3dd_4acc_aab3_87ee12191955.slice/crio-435af5eb31c71ec83140ec2a05aabc9f4084f27f1bc7a145fe3078de5088646c WatchSource:0}: Error finding container 435af5eb31c71ec83140ec2a05aabc9f4084f27f1bc7a145fe3078de5088646c: Status 404 returned error can't find the container with id 435af5eb31c71ec83140ec2a05aabc9f4084f27f1bc7a145fe3078de5088646c Nov 01 21:49:50.396758 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.396731 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 21:49:50.399396 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:49:50.399374 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df3d25d_99a3_4171_8ba5_42594ab44775.slice/crio-943014618cac1a4263762b9cbf8e1f894b25e490847b027f3c6b434855eea9ea WatchSource:0}: Error finding container 943014618cac1a4263762b9cbf8e1f894b25e490847b027f3c6b434855eea9ea: Status 404 returned error can't find the container with id 943014618cac1a4263762b9cbf8e1f894b25e490847b027f3c6b434855eea9ea Nov 01 21:49:50.509655 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.509618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerStarted","Data":"943014618cac1a4263762b9cbf8e1f894b25e490847b027f3c6b434855eea9ea"} Nov 01 21:49:50.510856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:50.510828 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-sk98t" event={"ID":"c842afcd-e3dd-4acc-aab3-87ee12191955","Type":"ContainerStarted","Data":"435af5eb31c71ec83140ec2a05aabc9f4084f27f1bc7a145fe3078de5088646c"} Nov 01 21:49:52.305784 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:49:52.305743 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:49:53.528332 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.528244 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-sk98t" event={"ID":"c842afcd-e3dd-4acc-aab3-87ee12191955","Type":"ContainerStarted","Data":"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d"} Nov 01 21:49:53.528760 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.528400 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:49:53.529856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.529831 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerStarted","Data":"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5"} Nov 01 21:49:53.529970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.529882 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:49:53.557668 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.557621 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-sk98t" podStartSLOduration=1.575715414 podStartE2EDuration="4.557609983s" totalImagesPullingTime="2.981894569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:53.556343491 +0000 UTC m=+340.530246028" watchObservedRunningTime="2026-11-01 21:49:53.557609983 +0000 UTC m=+340.531512522" Nov 01 21:49:53.632018 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:49:53.631963 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56cb88789f-6nn87" podStartSLOduration=1.770731875 podStartE2EDuration="4.631946288s" totalImagesPullingTime="2.861214413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:53.631355806 +0000 UTC m=+340.605258345" watchObservedRunningTime="2026-11-01 21:49:53.631946288 +0000 UTC m=+340.605848854" Nov 01 21:50:04.536490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.536450 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:50:04.536869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.536551 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:50:04.615884 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.615854 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:50:04.616125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.616077 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-sk98t" podUID="c842afcd-e3dd-4acc-aab3-87ee12191955" containerName="manager" containerID="cri-o://990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d" gracePeriod=30 Nov 01 21:50:04.855621 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.855597 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:50:04.983387 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.983354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\") pod \"c842afcd-e3dd-4acc-aab3-87ee12191955\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " Nov 01 21:50:04.983387 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.983403 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\" (UniqueName: \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\") pod \"c842afcd-e3dd-4acc-aab3-87ee12191955\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " Nov 01 21:50:04.983602 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.983432 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\") pod \"c842afcd-e3dd-4acc-aab3-87ee12191955\" (UID: \"c842afcd-e3dd-4acc-aab3-87ee12191955\") " Nov 01 21:50:04.985906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.985877 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs" pod "c842afcd-e3dd-4acc-aab3-87ee12191955" (UID: "c842afcd-e3dd-4acc-aab3-87ee12191955"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:04.986050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.985899 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c" pod "c842afcd-e3dd-4acc-aab3-87ee12191955" (UID: "c842afcd-e3dd-4acc-aab3-87ee12191955"). InnerVolumeSpecName "kube-api-access-pnr4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:04.986050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:04.985901 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert" pod "c842afcd-e3dd-4acc-aab3-87ee12191955" (UID: "c842afcd-e3dd-4acc-aab3-87ee12191955"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:05.084265 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.084233 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:05.084265 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.084257 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnr4c\" (UniqueName: \"kubernetes.io/projected/c842afcd-e3dd-4acc-aab3-87ee12191955-kube-api-access-pnr4c\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:05.084265 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.084267 4395 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c842afcd-e3dd-4acc-aab3-87ee12191955-webhook-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:05.577356 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.577320 4395 generic.go:356] "Generic (PLEG): container finished" podID="c842afcd-e3dd-4acc-aab3-87ee12191955" containerID="990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d" exitCode=0 Nov 01 21:50:05.577356 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.577360 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-sk98t" event={"ID":"c842afcd-e3dd-4acc-aab3-87ee12191955","Type":"ContainerDied","Data":"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d"} Nov 01 21:50:05.577870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.577367 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-sk98t" Nov 01 21:50:05.577870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.577409 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-sk98t" event={"ID":"c842afcd-e3dd-4acc-aab3-87ee12191955","Type":"ContainerDied","Data":"435af5eb31c71ec83140ec2a05aabc9f4084f27f1bc7a145fe3078de5088646c"} Nov 01 21:50:05.577870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.577437 4395 scope.go:122] "RemoveContainer" containerID="990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d" Nov 01 21:50:05.586257 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.586234 4395 scope.go:122] "RemoveContainer" containerID="990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d" Nov 01 21:50:05.586538 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:05.586519 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d\": container with ID starting with 990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d not found: ID does not exist" containerID="990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d" Nov 01 21:50:05.586590 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.586549 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d"} err="failed to get container status \"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d\": rpc error: code = NotFound desc = could not find container \"990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d\": container with ID starting with 990dee02d9bed40ccb84d3f7e737303e61b2a94cbea42dd591f0cc0413e6845d not found: ID does not exist" Nov 01 21:50:05.649964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.649921 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:50:05.663526 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:05.663497 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-sk98t"] Nov 01 21:50:06.753598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.753560 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2c8m7"] Nov 01 21:50:06.753990 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.753862 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c842afcd-e3dd-4acc-aab3-87ee12191955" containerName="manager" Nov 01 21:50:06.753990 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.753872 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c842afcd-e3dd-4acc-aab3-87ee12191955" containerName="manager" Nov 01 21:50:06.753990 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.753929 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c842afcd-e3dd-4acc-aab3-87ee12191955" containerName="manager" Nov 01 21:50:06.758228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.758203 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:06.764226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.764201 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v85mr\"" Nov 01 21:50:06.769557 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.769535 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2c8m7"] Nov 01 21:50:06.801575 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.799366 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xgv6s"] Nov 01 21:50:06.806699 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.806669 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 21:50:06.807221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.807199 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:06.814735 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.814699 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xgv6s"] Nov 01 21:50:06.830794 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.830773 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-whf42\"" Nov 01 21:50:06.830955 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.830894 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 21:50:06.899764 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.899737 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b03aef49-9d21-48e1-9f0f-71a95a6b5451-cert\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:06.899891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.899775 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxkr\" (UniqueName: \"kubernetes.io/projected/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-kube-api-access-dvxkr\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:06.899891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.899805 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-tls-certs\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:06.899891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:06.899863 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4t9l\" (UniqueName: \"kubernetes.io/projected/b03aef49-9d21-48e1-9f0f-71a95a6b5451-kube-api-access-z4t9l\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.001077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.001046 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b03aef49-9d21-48e1-9f0f-71a95a6b5451-cert\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.001214 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.001094 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxkr\" (UniqueName: \"kubernetes.io/projected/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-kube-api-access-dvxkr\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:07.001214 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.001120 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-tls-certs\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:07.001214 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.001142 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4t9l\" (UniqueName: \"kubernetes.io/projected/b03aef49-9d21-48e1-9f0f-71a95a6b5451-kube-api-access-z4t9l\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.003586 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.003555 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b03aef49-9d21-48e1-9f0f-71a95a6b5451-cert\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.003748 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.003710 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-tls-certs\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:07.013334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.013283 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxkr\" (UniqueName: \"kubernetes.io/projected/559bf8dc-f0e9-4069-aecc-b0d8de5e4111-kube-api-access-dvxkr\") pod \"model-serving-api-75bd89fdfc-xgv6s\" (UID: \"559bf8dc-f0e9-4069-aecc-b0d8de5e4111\") " pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:07.013434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.013285 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4t9l\" (UniqueName: \"kubernetes.io/projected/b03aef49-9d21-48e1-9f0f-71a95a6b5451-kube-api-access-z4t9l\") pod \"kserve-controller-manager-776cc66d47-2c8m7\" (UID: \"b03aef49-9d21-48e1-9f0f-71a95a6b5451\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.051368 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.051347 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4"] Nov 01 21:50:07.054802 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.054786 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.071070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.071019 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4"] Nov 01 21:50:07.071184 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.071135 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:07.123050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.122997 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:07.166347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.166310 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c842afcd-e3dd-4acc-aab3-87ee12191955" path="/var/lib/kubelet/pods/c842afcd-e3dd-4acc-aab3-87ee12191955/volumes" Nov 01 21:50:07.203560 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203587 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203721 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203688 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203768 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203726 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203768 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203751 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203785 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6cc\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-kube-api-access-hl6cc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.203853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.203832 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304578 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304623 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304642 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304657 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6cc\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-kube-api-access-hl6cc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304758 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.304978 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.304899 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.305513 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.305484 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.307041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.307007 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.307115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.307085 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.307552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.307537 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.307595 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.307572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.318364 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.318340 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.319082 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.319063 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6cc\" (UniqueName: \"kubernetes.io/projected/f7db28f1-af5a-43b6-a5b9-68183ebbb8b4-kube-api-access-hl6cc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v64b4\" (UID: \"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.347728 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.347634 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2c8m7"] Nov 01 21:50:07.350281 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:50:07.350261 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03aef49_9d21_48e1_9f0f_71a95a6b5451.slice/crio-878545f6d7d70412509e7a1f391d1f81fec1c838ea063dc95837ba0448b6b1f0 WatchSource:0}: Error finding container 878545f6d7d70412509e7a1f391d1f81fec1c838ea063dc95837ba0448b6b1f0: Status 404 returned error can't find the container with id 878545f6d7d70412509e7a1f391d1f81fec1c838ea063dc95837ba0448b6b1f0 Nov 01 21:50:07.366354 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.366337 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:07.407467 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:50:07.407439 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559bf8dc_f0e9_4069_aecc_b0d8de5e4111.slice/crio-de28a87504858efe5964b1d85f2d8a718bf5b5b453cc173aa2641d8bc39104f1 WatchSource:0}: Error finding container de28a87504858efe5964b1d85f2d8a718bf5b5b453cc173aa2641d8bc39104f1: Status 404 returned error can't find the container with id de28a87504858efe5964b1d85f2d8a718bf5b5b453cc173aa2641d8bc39104f1 Nov 01 21:50:07.407717 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.407694 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xgv6s"] Nov 01 21:50:07.587518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.587427 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" event={"ID":"b03aef49-9d21-48e1-9f0f-71a95a6b5451","Type":"ContainerStarted","Data":"878545f6d7d70412509e7a1f391d1f81fec1c838ea063dc95837ba0448b6b1f0"} Nov 01 21:50:07.588604 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.588578 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" event={"ID":"559bf8dc-f0e9-4069-aecc-b0d8de5e4111","Type":"ContainerStarted","Data":"de28a87504858efe5964b1d85f2d8a718bf5b5b453cc173aa2641d8bc39104f1"} Nov 01 21:50:07.644583 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.644558 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4"] Nov 01 21:50:07.647231 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:50:07.647210 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7db28f1_af5a_43b6_a5b9_68183ebbb8b4.slice/crio-ce7a0d5a9f82429e775d3b7d3af634ca9f52047aba9a328cf8b9bca741a2110b WatchSource:0}: Error finding container ce7a0d5a9f82429e775d3b7d3af634ca9f52047aba9a328cf8b9bca741a2110b: Status 404 returned error can't find the container with id ce7a0d5a9f82429e775d3b7d3af634ca9f52047aba9a328cf8b9bca741a2110b Nov 01 21:50:07.649172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.649144 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:50:07.649255 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:07.649199 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:50:08.597984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.597215 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" event={"ID":"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4","Type":"ContainerStarted","Data":"c74f0bc0840c24ced1a43c681cb3a8fdbd67e1c76fdf01fec621acd061cf9472"} Nov 01 21:50:08.597984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.597285 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" event={"ID":"f7db28f1-af5a-43b6-a5b9-68183ebbb8b4","Type":"ContainerStarted","Data":"ce7a0d5a9f82429e775d3b7d3af634ca9f52047aba9a328cf8b9bca741a2110b"} Nov 01 21:50:08.597984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.597640 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:08.600555 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.600526 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" Nov 01 21:50:08.624274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.624221 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v64b4" podStartSLOduration=1.624199618 podStartE2EDuration="1.624199618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:08.622185735 +0000 UTC m=+355.596088292" watchObservedRunningTime="2026-11-01 21:50:08.624199618 +0000 UTC m=+355.598102160" Nov 01 21:50:08.677599 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.677562 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:50:08.678250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.677808 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" containerID="cri-o://91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3" gracePeriod=30 Nov 01 21:50:08.964930 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:08.964909 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:50:09.123529 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123448 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123687 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123547 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123687 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123640 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123687 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123673 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123854 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123735 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123854 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123788 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.123854 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.123821 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\") pod \"7e8a0748-c612-44d3-ad7b-303f948ce09b\" (UID: \"7e8a0748-c612-44d3-ad7b-303f948ce09b\") " Nov 01 21:50:09.125173 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.125141 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:50:09.127167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.127124 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:50:09.130355 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.130326 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:09.130355 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.130341 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:09.130511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.130359 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "kube-api-access-gbvxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:09.131988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.131961 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:09.143365 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.138510 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts" pod "7e8a0748-c612-44d3-ad7b-303f948ce09b" (UID: "7e8a0748-c612-44d3-ad7b-303f948ce09b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:09.190701 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:09.190665 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8a0748_c612_44d3_ad7b_303f948ce09b.slice\": RecentStats: unable to find data in memory cache]" Nov 01 21:50:09.224530 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224495 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-dns-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224534 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbvxh\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-kube-api-access-gbvxh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224551 4395 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-cacerts\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224565 4395 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-kubeconfig\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224576 4395 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e8a0748-c612-44d3-ad7b-303f948ce09b-local-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224584 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.224675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.224592 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e8a0748-c612-44d3-ad7b-303f948ce09b-istio-csr-ca-configmap\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:09.604362 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.604321 4395 generic.go:356] "Generic (PLEG): container finished" podID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerID="91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3" exitCode=0 Nov 01 21:50:09.604912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.604397 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" Nov 01 21:50:09.604912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.604414 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" event={"ID":"7e8a0748-c612-44d3-ad7b-303f948ce09b","Type":"ContainerDied","Data":"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3"} Nov 01 21:50:09.604912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.604498 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt" event={"ID":"7e8a0748-c612-44d3-ad7b-303f948ce09b","Type":"ContainerDied","Data":"66fe64e3376ee2e0121c2124941042128c8b075fa1ab177d547ea9dc8a22cca1"} Nov 01 21:50:09.604912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.604524 4395 scope.go:122] "RemoveContainer" containerID="91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3" Nov 01 21:50:09.617235 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.617215 4395 scope.go:122] "RemoveContainer" containerID="91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3" Nov 01 21:50:09.617644 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:09.617618 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3\": container with ID starting with 91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3 not found: ID does not exist" containerID="91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3" Nov 01 21:50:09.617787 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.617656 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3"} err="failed to get container status \"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3\": rpc error: code = NotFound desc = could not find container \"91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3\": container with ID starting with 91e26abc30184d988eb19531d8e1ebe32d5e366b77c730d400e2c7a22764a1c3 not found: ID does not exist" Nov 01 21:50:09.624938 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.624911 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:50:09.629063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:09.629041 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zpvwt"] Nov 01 21:50:10.610467 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:10.610434 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" event={"ID":"559bf8dc-f0e9-4069-aecc-b0d8de5e4111","Type":"ContainerStarted","Data":"bf9948e45a46eb057fb3774b583d8b7e1416b4e6a5a0889d58ac9ba57beefbee"} Nov 01 21:50:10.610881 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:10.610507 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:10.613011 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:10.612987 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" event={"ID":"b03aef49-9d21-48e1-9f0f-71a95a6b5451","Type":"ContainerStarted","Data":"42e28a12cff9d0ab07935bc4a1278dc37e7d670fae20ecc737b17d315ef5ee82"} Nov 01 21:50:10.629752 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:10.629711 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" podStartSLOduration=2.430004308 podStartE2EDuration="4.629699078s" totalImagesPullingTime="2.19969477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:10.628776398 +0000 UTC m=+357.602678943" watchObservedRunningTime="2026-11-01 21:50:10.629699078 +0000 UTC m=+357.603601614" Nov 01 21:50:10.647454 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:10.647403 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" podStartSLOduration=2.069878564 podStartE2EDuration="4.647390809s" totalImagesPullingTime="2.577512245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:10.646976506 +0000 UTC m=+357.620879052" watchObservedRunningTime="2026-11-01 21:50:10.647390809 +0000 UTC m=+357.621293348" Nov 01 21:50:11.162833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:11.162793 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" path="/var/lib/kubelet/pods/7e8a0748-c612-44d3-ad7b-303f948ce09b/volumes" Nov 01 21:50:11.617683 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:11.617643 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:21.621970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:21.621942 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xgv6s" Nov 01 21:50:22.305010 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:22.304976 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:50:25.107138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.107103 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:25.107497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.107469 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" Nov 01 21:50:25.107497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.107481 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" Nov 01 21:50:25.107570 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.107548 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8a0748-c612-44d3-ad7b-303f948ce09b" containerName="discovery" Nov 01 21:50:25.110376 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.110357 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.116444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.116421 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 21:50:25.116554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.116464 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zltrj\"" Nov 01 21:50:25.116554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.116504 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 21:50:25.123266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.123242 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:25.142428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.142399 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.142857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.142455 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.142857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.142482 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.142857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.142509 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5477b\" (UniqueName: \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.243017 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.242990 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.243187 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.243055 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.243187 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.243087 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5477b\" (UniqueName: \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.243311 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.243254 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.243695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.243670 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.245640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.245620 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.245735 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.245682 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.255424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.255401 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5477b\" (UniqueName: \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\") pod \"maas-api-6d669c7fcd-7hsnd\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.423807 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.423731 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:25.698542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:25.698466 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:25.701350 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:50:25.701325 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732f5f4e_f838_48a0_abba_22fd03f6e235.slice/crio-76f13d242efe3ba4b1ffac616a7965915028e2fc38d16b4aa6380f7f5bc42d77 WatchSource:0}: Error finding container 76f13d242efe3ba4b1ffac616a7965915028e2fc38d16b4aa6380f7f5bc42d77: Status 404 returned error can't find the container with id 76f13d242efe3ba4b1ffac616a7965915028e2fc38d16b4aa6380f7f5bc42d77 Nov 01 21:50:26.684998 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:26.684955 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" event={"ID":"732f5f4e-f838-48a0-abba-22fd03f6e235","Type":"ContainerStarted","Data":"76f13d242efe3ba4b1ffac616a7965915028e2fc38d16b4aa6380f7f5bc42d77"} Nov 01 21:50:27.690965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:27.690885 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" event={"ID":"732f5f4e-f838-48a0-abba-22fd03f6e235","Type":"ContainerStarted","Data":"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83"} Nov 01 21:50:27.691353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:27.691021 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:27.711799 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:27.711745 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" podStartSLOduration=1.115328226 podStartE2EDuration="2.711731065s" totalImagesPullingTime="1.596402839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:27.710779024 +0000 UTC m=+374.684681588" watchObservedRunningTime="2026-11-01 21:50:27.711731065 +0000 UTC m=+374.685633605" Nov 01 21:50:33.701349 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:33.701312 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:42.627365 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:42.627335 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2c8m7" Nov 01 21:50:44.400811 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.400776 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:44.401305 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.400981 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" podUID="732f5f4e-f838-48a0-abba-22fd03f6e235" containerName="maas-api" containerID="cri-o://461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83" gracePeriod=30 Nov 01 21:50:44.635449 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.635423 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:44.712726 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.712648 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\") pod \"732f5f4e-f838-48a0-abba-22fd03f6e235\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " Nov 01 21:50:44.712726 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.712719 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\") pod \"732f5f4e-f838-48a0-abba-22fd03f6e235\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " Nov 01 21:50:44.712928 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.712819 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\" (UniqueName: \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\") pod \"732f5f4e-f838-48a0-abba-22fd03f6e235\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " Nov 01 21:50:44.712928 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.712888 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\") pod \"732f5f4e-f838-48a0-abba-22fd03f6e235\" (UID: \"732f5f4e-f838-48a0-abba-22fd03f6e235\") " Nov 01 21:50:44.713100 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.713072 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca" pod "732f5f4e-f838-48a0-abba-22fd03f6e235" (UID: "732f5f4e-f838-48a0-abba-22fd03f6e235"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:50:44.713210 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.713193 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732f5f4e-f838-48a0-abba-22fd03f6e235-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:44.715070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.715000 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b" pod "732f5f4e-f838-48a0-abba-22fd03f6e235" (UID: "732f5f4e-f838-48a0-abba-22fd03f6e235"). InnerVolumeSpecName "kube-api-access-5477b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:50:44.715070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.715041 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs" pod "732f5f4e-f838-48a0-abba-22fd03f6e235" (UID: "732f5f4e-f838-48a0-abba-22fd03f6e235"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:44.715216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.715114 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls" pod "732f5f4e-f838-48a0-abba-22fd03f6e235" (UID: "732f5f4e-f838-48a0-abba-22fd03f6e235"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:50:44.761110 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.761081 4395 generic.go:356] "Generic (PLEG): container finished" podID="732f5f4e-f838-48a0-abba-22fd03f6e235" containerID="461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83" exitCode=0 Nov 01 21:50:44.761231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.761114 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" event={"ID":"732f5f4e-f838-48a0-abba-22fd03f6e235","Type":"ContainerDied","Data":"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83"} Nov 01 21:50:44.761231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.761133 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" Nov 01 21:50:44.761231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.761156 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd" event={"ID":"732f5f4e-f838-48a0-abba-22fd03f6e235","Type":"ContainerDied","Data":"76f13d242efe3ba4b1ffac616a7965915028e2fc38d16b4aa6380f7f5bc42d77"} Nov 01 21:50:44.761231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.761177 4395 scope.go:122] "RemoveContainer" containerID="461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83" Nov 01 21:50:44.771100 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.771062 4395 scope.go:122] "RemoveContainer" containerID="461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83" Nov 01 21:50:44.771386 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:44.771368 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83\": container with ID starting with 461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83 not found: ID does not exist" containerID="461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83" Nov 01 21:50:44.771433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.771397 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83"} err="failed to get container status \"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83\": rpc error: code = NotFound desc = could not find container \"461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83\": container with ID starting with 461f051609804dca50e3710ccf9167c4840567de4c41c1744049831807a5ec83 not found: ID does not exist" Nov 01 21:50:44.788048 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.788006 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:44.792803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.792780 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-7hsnd"] Nov 01 21:50:44.814583 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.814553 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:44.814583 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.814581 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5477b\" (UniqueName: \"kubernetes.io/projected/732f5f4e-f838-48a0-abba-22fd03f6e235-kube-api-access-5477b\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:44.814768 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:44.814592 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732f5f4e-f838-48a0-abba-22fd03f6e235-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:50:45.165200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:50:45.165164 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732f5f4e-f838-48a0-abba-22fd03f6e235" path="/var/lib/kubelet/pods/732f5f4e-f838-48a0-abba-22fd03f6e235/volumes" Nov 01 21:50:52.299813 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:50:52.299779 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:51:22.300128 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:51:22.300094 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:51:52.301663 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:51:52.301618 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:52:11.207817 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.207774 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk"] Nov 01 21:52:11.208398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.208117 4395 cpu_manager.go:414] "containerMap: removing container" podUID="732f5f4e-f838-48a0-abba-22fd03f6e235" containerName="maas-api" Nov 01 21:52:11.208398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.208129 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732f5f4e-f838-48a0-abba-22fd03f6e235" containerName="maas-api" Nov 01 21:52:11.208398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.208188 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="732f5f4e-f838-48a0-abba-22fd03f6e235" containerName="maas-api" Nov 01 21:52:11.211151 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.211127 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.220476 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.220451 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 21:52:11.220710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.220678 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rfgwk\"" Nov 01 21:52:11.221808 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.221779 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 21:52:11.225451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.225427 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 21:52:11.228675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.228308 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk"] Nov 01 21:52:11.286745 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.286723 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.286861 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.286785 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.286955 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.286935 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6p2\" (UniqueName: \"kubernetes.io/projected/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-kube-api-access-jp6p2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.287007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.286969 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.287068 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.287044 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.287103 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.287073 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.387862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.387836 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.387963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.387899 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.387963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.387934 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.387960 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6p2\" (UniqueName: \"kubernetes.io/projected/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-kube-api-access-jp6p2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.387988 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.388012 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388316 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.388295 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388392 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.388334 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.388451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.388395 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.390328 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.390308 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.390491 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.390475 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.400135 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.400115 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6p2\" (UniqueName: \"kubernetes.io/projected/1429c5b6-feda-4fa4-8fd9-f041fe7f98ef-kube-api-access-jp6p2\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk\" (UID: \"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.528077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.528052 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:11.807543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:11.807417 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk"] Nov 01 21:52:11.810268 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:11.810241 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1429c5b6_feda_4fa4_8fd9_f041fe7f98ef.slice/crio-0f8e2d98629b3f20adf4b3285daeec9a46afd45b916f644e01622545d0b67a9e WatchSource:0}: Error finding container 0f8e2d98629b3f20adf4b3285daeec9a46afd45b916f644e01622545d0b67a9e: Status 404 returned error can't find the container with id 0f8e2d98629b3f20adf4b3285daeec9a46afd45b916f644e01622545d0b67a9e Nov 01 21:52:12.126046 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:12.125949 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" event={"ID":"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef","Type":"ContainerStarted","Data":"0f8e2d98629b3f20adf4b3285daeec9a46afd45b916f644e01622545d0b67a9e"} Nov 01 21:52:13.218907 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.218876 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7"] Nov 01 21:52:13.222330 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.222308 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.227930 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.227906 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 21:52:13.233381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.233358 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7"] Nov 01 21:52:13.306645 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306616 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.306749 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306664 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpvg\" (UniqueName: \"kubernetes.io/projected/79984a24-8785-4acc-8259-cc1e4f908fb5-kube-api-access-gbpvg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.306804 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306772 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79984a24-8785-4acc-8259-cc1e4f908fb5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.306838 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.306883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306869 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.306984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.306964 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407573 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407611 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpvg\" (UniqueName: \"kubernetes.io/projected/79984a24-8785-4acc-8259-cc1e4f908fb5-kube-api-access-gbpvg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79984a24-8785-4acc-8259-cc1e4f908fb5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407653 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407673 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.407859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.407714 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.408093 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.408010 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.408160 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.408118 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.408215 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.408176 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.410077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.410055 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79984a24-8785-4acc-8259-cc1e4f908fb5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.410253 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.410238 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79984a24-8785-4acc-8259-cc1e4f908fb5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.419606 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.419587 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpvg\" (UniqueName: \"kubernetes.io/projected/79984a24-8785-4acc-8259-cc1e4f908fb5-kube-api-access-gbpvg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-svbr7\" (UID: \"79984a24-8785-4acc-8259-cc1e4f908fb5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.538109 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.538084 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:13.812001 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:13.811920 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7"] Nov 01 21:52:13.816290 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:13.816250 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79984a24_8785_4acc_8259_cc1e4f908fb5.slice/crio-f9ad2eb794e43e38ab113513fbe7daaf2166aed641308a4b03f5d4eb471515fc WatchSource:0}: Error finding container f9ad2eb794e43e38ab113513fbe7daaf2166aed641308a4b03f5d4eb471515fc: Status 404 returned error can't find the container with id f9ad2eb794e43e38ab113513fbe7daaf2166aed641308a4b03f5d4eb471515fc Nov 01 21:52:14.084204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.084139 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk"] Nov 01 21:52:14.087374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.087352 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.091167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.091145 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 21:52:14.097777 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.097758 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk"] Nov 01 21:52:14.135549 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.135515 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" event={"ID":"1429c5b6-feda-4fa4-8fd9-f041fe7f98ef","Type":"ContainerStarted","Data":"99059e4dc10372d4f9d8e862c8ce36d48ca01d2ed02d52f46946f20d5ae3534b"} Nov 01 21:52:14.137061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.137012 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" event={"ID":"79984a24-8785-4acc-8259-cc1e4f908fb5","Type":"ContainerStarted","Data":"f9ad2eb794e43e38ab113513fbe7daaf2166aed641308a4b03f5d4eb471515fc"} Nov 01 21:52:14.157007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.156965 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" podStartSLOduration=1.851452128 podStartE2EDuration="3.156951805s" totalImagesPullingTime="1.305499677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:14.155272748 +0000 UTC m=+481.129175301" watchObservedRunningTime="2026-11-01 21:52:14.156951805 +0000 UTC m=+481.130854344" Nov 01 21:52:14.214581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214553 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.214713 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214599 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.214713 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214624 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3febf414-14ee-4545-abfa-1f421ad43f31-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.214823 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214732 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.214823 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214775 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.214823 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.214790 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsjs\" (UniqueName: \"kubernetes.io/projected/3febf414-14ee-4545-abfa-1f421ad43f31-kube-api-access-tpsjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316298 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316342 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3febf414-14ee-4545-abfa-1f421ad43f31-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316414 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316505 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316550 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsjs\" (UniqueName: \"kubernetes.io/projected/3febf414-14ee-4545-abfa-1f421ad43f31-kube-api-access-tpsjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.316769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316710 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.317077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.316877 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.317077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.317010 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.318852 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.318831 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3febf414-14ee-4545-abfa-1f421ad43f31-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.319152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.319135 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3febf414-14ee-4545-abfa-1f421ad43f31-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.328870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.328839 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsjs\" (UniqueName: \"kubernetes.io/projected/3febf414-14ee-4545-abfa-1f421ad43f31-kube-api-access-tpsjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2mhpk\" (UID: \"3febf414-14ee-4545-abfa-1f421ad43f31\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.404901 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.404845 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:14.528719 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.528694 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:14.678533 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:14.678473 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk"] Nov 01 21:52:14.681170 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:14.681142 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3febf414_14ee_4545_abfa_1f421ad43f31.slice/crio-458b33df8aad9123b2b70d6ea2a07035501967654d2b9726f03eacb14139deab WatchSource:0}: Error finding container 458b33df8aad9123b2b70d6ea2a07035501967654d2b9726f03eacb14139deab: Status 404 returned error can't find the container with id 458b33df8aad9123b2b70d6ea2a07035501967654d2b9726f03eacb14139deab Nov 01 21:52:15.143920 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.143883 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" event={"ID":"79984a24-8785-4acc-8259-cc1e4f908fb5","Type":"ContainerStarted","Data":"a9a420729e775e6df999d0ab7bb007f8e84c7aad65243a05106c5272411527f6"} Nov 01 21:52:15.145542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.145508 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" event={"ID":"3febf414-14ee-4545-abfa-1f421ad43f31","Type":"ContainerStarted","Data":"6949952c101b2f6725a820b6b568d7da9ee5db3eea82ba6f9b909d253b735202"} Nov 01 21:52:15.145666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.145557 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" event={"ID":"3febf414-14ee-4545-abfa-1f421ad43f31","Type":"ContainerStarted","Data":"458b33df8aad9123b2b70d6ea2a07035501967654d2b9726f03eacb14139deab"} Nov 01 21:52:15.167792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.167753 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" podStartSLOduration=1.895688103 podStartE2EDuration="2.167730853s" totalImagesPullingTime="272.04275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:15.166256477 +0000 UTC m=+482.140159006" watchObservedRunningTime="2026-11-01 21:52:15.167730853 +0000 UTC m=+482.141633428" Nov 01 21:52:15.186496 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.186457 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" podStartSLOduration=0.894454079 podStartE2EDuration="1.186445699s" totalImagesPullingTime="291.99162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:15.185820857 +0000 UTC m=+482.159723397" watchObservedRunningTime="2026-11-01 21:52:15.186445699 +0000 UTC m=+482.160348237" Nov 01 21:52:15.405696 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.405671 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:15.538631 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:15.538601 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:18.706626 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.706592 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64"] Nov 01 21:52:18.709932 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.709914 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.714730 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.714698 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 21:52:18.721534 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.721510 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64"] Nov 01 21:52:18.852064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852016 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.852216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852109 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.852216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852132 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qd6w\" (UniqueName: \"kubernetes.io/projected/a389fbce-17d4-4613-884d-37aa85377992-kube-api-access-7qd6w\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.852216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852196 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.852398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852244 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389fbce-17d4-4613-884d-37aa85377992-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.852398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.852354 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953140 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953181 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953205 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qd6w\" (UniqueName: \"kubernetes.io/projected/a389fbce-17d4-4613-884d-37aa85377992-kube-api-access-7qd6w\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953229 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953258 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389fbce-17d4-4613-884d-37aa85377992-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953400 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953619 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953596 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953685 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953606 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.953740 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.953703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.955666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.955636 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a389fbce-17d4-4613-884d-37aa85377992-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.956012 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.955992 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a389fbce-17d4-4613-884d-37aa85377992-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:18.965600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:18.965547 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qd6w\" (UniqueName: \"kubernetes.io/projected/a389fbce-17d4-4613-884d-37aa85377992-kube-api-access-7qd6w\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64\" (UID: \"a389fbce-17d4-4613-884d-37aa85377992\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:19.023258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:19.023235 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:19.301314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:19.301270 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64"] Nov 01 21:52:19.303852 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:19.303826 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda389fbce_17d4_4613_884d_37aa85377992.slice/crio-3e2f63600926e6ed5dcbef0410e46c208f32feb5d0446451d08f76d9c345359d WatchSource:0}: Error finding container 3e2f63600926e6ed5dcbef0410e46c208f32feb5d0446451d08f76d9c345359d: Status 404 returned error can't find the container with id 3e2f63600926e6ed5dcbef0410e46c208f32feb5d0446451d08f76d9c345359d Nov 01 21:52:20.168367 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:20.168333 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" event={"ID":"a389fbce-17d4-4613-884d-37aa85377992","Type":"ContainerStarted","Data":"64da08adc0d5fa3f5ca4fe610748d7d248b883f10fdacb6314cd7331e9973f98"} Nov 01 21:52:20.168719 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:20.168379 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" event={"ID":"a389fbce-17d4-4613-884d-37aa85377992","Type":"ContainerStarted","Data":"3e2f63600926e6ed5dcbef0410e46c208f32feb5d0446451d08f76d9c345359d"} Nov 01 21:52:20.189314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:20.189268 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" podStartSLOduration=1.920871352 podStartE2EDuration="2.189253962s" totalImagesPullingTime="268.38261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:20.18772239 +0000 UTC m=+487.161624977" watchObservedRunningTime="2026-11-01 21:52:20.189253962 +0000 UTC m=+487.163156501" Nov 01 21:52:21.023750 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:21.023717 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:21.528818 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:21.528795 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:21.541148 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:21.541124 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:22.188158 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:22.188121 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" Nov 01 21:52:22.301755 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:52:22.301726 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:52:23.182197 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:23.182166 4395 generic.go:356] "Generic (PLEG): container finished" podID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerID="ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5" exitCode=1 Nov 01 21:52:23.182661 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:23.182243 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerDied","Data":"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5"} Nov 01 21:52:23.182661 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:23.182620 4395 scope.go:122] "RemoveContainer" containerID="ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5" Nov 01 21:52:23.539193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:23.539162 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:23.551506 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:23.551486 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:24.188829 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:24.188733 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerStarted","Data":"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c"} Nov 01 21:52:24.189470 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:24.189447 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:52:24.200937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:24.200916 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" Nov 01 21:52:24.405147 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:24.405073 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:24.417474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:24.417453 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:25.204476 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:25.204444 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" Nov 01 21:52:29.024328 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:29.024298 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:29.036380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:29.036360 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:29.220367 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:29.220345 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" Nov 01 21:52:35.195631 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:35.195603 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 21:52:45.535611 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.535578 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:52:45.542455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.542433 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.549309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.549277 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:52:45.550677 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.550637 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 21:52:45.550835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.550679 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 21:52:45.550835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.550825 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6xtkv\"" Nov 01 21:52:45.646831 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.646805 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.646972 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.646876 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwlbr\" (UniqueName: \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.748140 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.748117 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwlbr\" (UniqueName: \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.748248 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.748159 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.750750 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.750728 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.760670 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.760650 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwlbr\" (UniqueName: \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\") pod \"ai-gateway-controller-7d6c696b49-l9994\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:45.857595 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:45.857546 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:46.133043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:46.133005 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:52:46.135383 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:46.135360 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1636fe18_e78a_46e5_b2ae_00b780e26d7e.slice/crio-4a71ab5353fce6e37685092c1140e126d33ef2fa8a7994f6a67aea8cc03fd636 WatchSource:0}: Error finding container 4a71ab5353fce6e37685092c1140e126d33ef2fa8a7994f6a67aea8cc03fd636: Status 404 returned error can't find the container with id 4a71ab5353fce6e37685092c1140e126d33ef2fa8a7994f6a67aea8cc03fd636 Nov 01 21:52:46.278769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:46.278739 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" event={"ID":"1636fe18-e78a-46e5-b2ae-00b780e26d7e","Type":"ContainerStarted","Data":"4a71ab5353fce6e37685092c1140e126d33ef2fa8a7994f6a67aea8cc03fd636"} Nov 01 21:52:51.300430 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:51.300397 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" event={"ID":"1636fe18-e78a-46e5-b2ae-00b780e26d7e","Type":"ContainerStarted","Data":"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a"} Nov 01 21:52:51.300835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:51.300554 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:52:51.319804 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:51.319754 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" podStartSLOduration=1.838472186 podStartE2EDuration="6.319739386s" totalImagesPullingTime="4.4812672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:51.318923326 +0000 UTC m=+518.292825865" watchObservedRunningTime="2026-11-01 21:52:51.319739386 +0000 UTC m=+518.293641925" Nov 01 21:52:52.301439 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:52:52.301389 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:52:52.357098 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.357066 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 21:52:52.360792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.360769 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.368772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.368717 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tfhbq\"" Nov 01 21:52:52.379909 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.379882 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 21:52:52.399441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.399415 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.400200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.400179 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 21:52:52.402911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.402551 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 21:52:52.402911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.402904 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.402974 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403012 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403074 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403101 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403705 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403668 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95r4m\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403765 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.403948 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.403930 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.404116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.404095 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.492892 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.492862 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7"] Nov 01 21:52:52.496273 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.496253 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.504550 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504524 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.504665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504562 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.504665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504584 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.504665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504656 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.504840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504689 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxb55\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.504840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504717 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.504840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.504840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504828 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.505123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504845 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.505123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504858 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.505123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504882 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.505123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.504925 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.505430 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.505396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.505525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.505504 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.505650 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.505633 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 21:52:52.505757 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.505735 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x49lb\"" Nov 01 21:52:52.506203 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506179 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95r4m\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.506303 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506219 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.506303 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506246 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.506303 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506284 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.506434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506313 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.507134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506598 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.507134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506884 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.507134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.506991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.507134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.507095 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.508341 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.508317 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.508444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.508358 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7"] Nov 01 21:52:52.517545 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.517525 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.522601 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.522569 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95r4m\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.527381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.527362 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vktwt\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.528770 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.528745 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-tfddh"] Nov 01 21:52:52.535447 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.535426 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.540911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.540893 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-tfddh"] Nov 01 21:52:52.608122 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608016 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608122 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608090 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608121 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-plugins-config-volume\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.608309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608160 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48ace94-f380-4e38-917f-8f266e8ab437-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.608309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608255 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608296 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608313 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wdm\" (UniqueName: \"kubernetes.io/projected/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-kube-api-access-r5wdm\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.608509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608334 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608663 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608612 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608663 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608650 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608673 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608675 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608727 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xsl\" (UniqueName: \"kubernetes.io/projected/b48ace94-f380-4e38-917f-8f266e8ab437-kube-api-access-m7xsl\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.608968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608843 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608851 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxb55\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.608968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.608938 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.610846 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.610823 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.611236 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.611222 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.620856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.620832 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxb55\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.620964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.620946 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vwrt5\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.674866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.674828 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:52.709581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.709544 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wdm\" (UniqueName: \"kubernetes.io/projected/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-kube-api-access-r5wdm\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.709729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.709656 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xsl\" (UniqueName: \"kubernetes.io/projected/b48ace94-f380-4e38-917f-8f266e8ab437-kube-api-access-m7xsl\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.709729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.709708 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-plugins-config-volume\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.709885 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.709745 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48ace94-f380-4e38-917f-8f266e8ab437-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.710714 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.710692 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b48ace94-f380-4e38-917f-8f266e8ab437-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.710783 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.710699 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-plugins-config-volume\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.714266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.714244 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:52.721895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.721875 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wdm\" (UniqueName: \"kubernetes.io/projected/3db02aed-580e-42bc-88f7-9b7b06b6ce3b-kube-api-access-r5wdm\") pod \"payload-processing-7bb9489676-tfddh\" (UID: \"3db02aed-580e-42bc-88f7-9b7b06b6ce3b\") " pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.722262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.722243 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xsl\" (UniqueName: \"kubernetes.io/projected/b48ace94-f380-4e38-917f-8f266e8ab437-kube-api-access-m7xsl\") pod \"payload-pre-processing-7b98bb678c-dxpk7\" (UID: \"b48ace94-f380-4e38-917f-8f266e8ab437\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.810936 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.810909 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:52.857957 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.857925 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:52.961343 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:52.961318 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 21:52:52.963307 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:52.963281 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7126f5d1_347c_4da5_a744_226fa0eb4e10.slice/crio-1df8f4f3b32ec176b3acf6269c0be7b9b12d5fb59cf7286d53c602a25550b27f WatchSource:0}: Error finding container 1df8f4f3b32ec176b3acf6269c0be7b9b12d5fb59cf7286d53c602a25550b27f: Status 404 returned error can't find the container with id 1df8f4f3b32ec176b3acf6269c0be7b9b12d5fb59cf7286d53c602a25550b27f Nov 01 21:52:53.019513 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.019382 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 21:52:53.021683 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:53.021653 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128a87b7_b7dd_44b2_9ac8_4271d835e526.slice/crio-bee77fc8ccf38e6d6f920189f5488d28dc46dc1230601244c1797979e9dac658 WatchSource:0}: Error finding container bee77fc8ccf38e6d6f920189f5488d28dc46dc1230601244c1797979e9dac658: Status 404 returned error can't find the container with id bee77fc8ccf38e6d6f920189f5488d28dc46dc1230601244c1797979e9dac658 Nov 01 21:52:53.089908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.089876 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7"] Nov 01 21:52:53.092826 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:53.092787 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48ace94_f380_4e38_917f_8f266e8ab437.slice/crio-eca986c57be826e9994762919cc20b3f68423d3814f32678c14b823f92d64af5 WatchSource:0}: Error finding container eca986c57be826e9994762919cc20b3f68423d3814f32678c14b823f92d64af5: Status 404 returned error can't find the container with id eca986c57be826e9994762919cc20b3f68423d3814f32678c14b823f92d64af5 Nov 01 21:52:53.137085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.136888 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-tfddh"] Nov 01 21:52:53.139456 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:52:53.139430 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db02aed_580e_42bc_88f7_9b7b06b6ce3b.slice/crio-9985071b596c73799d5ec04509d11c4d9a43add23ed305e9f95d5d0150c234e8 WatchSource:0}: Error finding container 9985071b596c73799d5ec04509d11c4d9a43add23ed305e9f95d5d0150c234e8: Status 404 returned error can't find the container with id 9985071b596c73799d5ec04509d11c4d9a43add23ed305e9f95d5d0150c234e8 Nov 01 21:52:53.310840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.310767 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" event={"ID":"3db02aed-580e-42bc-88f7-9b7b06b6ce3b","Type":"ContainerStarted","Data":"9985071b596c73799d5ec04509d11c4d9a43add23ed305e9f95d5d0150c234e8"} Nov 01 21:52:53.312513 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.312484 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" event={"ID":"7126f5d1-347c-4da5-a744-226fa0eb4e10","Type":"ContainerStarted","Data":"1df8f4f3b32ec176b3acf6269c0be7b9b12d5fb59cf7286d53c602a25550b27f"} Nov 01 21:52:53.313921 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.313892 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" event={"ID":"128a87b7-b7dd-44b2-9ac8-4271d835e526","Type":"ContainerStarted","Data":"bee77fc8ccf38e6d6f920189f5488d28dc46dc1230601244c1797979e9dac658"} Nov 01 21:52:53.315140 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:53.315119 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" event={"ID":"b48ace94-f380-4e38-917f-8f266e8ab437","Type":"ContainerStarted","Data":"eca986c57be826e9994762919cc20b3f68423d3814f32678c14b823f92d64af5"} Nov 01 21:52:56.339008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.338913 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" event={"ID":"3db02aed-580e-42bc-88f7-9b7b06b6ce3b","Type":"ContainerStarted","Data":"61f6c0472952affc64b19aac146bb9da7e0bf8d20e093b151ad37d5ef4919544"} Nov 01 21:52:56.339008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.338989 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:52:56.340889 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.340859 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" event={"ID":"b48ace94-f380-4e38-917f-8f266e8ab437","Type":"ContainerStarted","Data":"23dd56245c0c328f857675c8abd93b98ec2c2f35bfd47547ce1444e4cf424046"} Nov 01 21:52:56.341014 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.341006 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:52:56.359145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.359102 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" podStartSLOduration=1.282270864 podStartE2EDuration="4.359089105s" totalImagesPullingTime="3.076818241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:56.357903738 +0000 UTC m=+523.331806278" watchObservedRunningTime="2026-11-01 21:52:56.359089105 +0000 UTC m=+523.332991645" Nov 01 21:52:56.378851 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.378799 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" podStartSLOduration=1.407067416 podStartE2EDuration="4.378776439s" totalImagesPullingTime="2.971709023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:56.376491232 +0000 UTC m=+523.350393772" watchObservedRunningTime="2026-11-01 21:52:56.378776439 +0000 UTC m=+523.352679003" Nov 01 21:52:56.832180 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.832143 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:56.832267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.832215 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:56.832267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:56.832247 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:57.166665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.166628 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:57.166764 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.166704 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:57.166764 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.166733 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:52:57.350794 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.350747 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" event={"ID":"7126f5d1-347c-4da5-a744-226fa0eb4e10","Type":"ContainerStarted","Data":"53d7bdd9c3b9a57f6a1c830beef6758880f5d4e8e3bc3467630baab029903d44"} Nov 01 21:52:57.352751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.352720 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" event={"ID":"128a87b7-b7dd-44b2-9ac8-4271d835e526","Type":"ContainerStarted","Data":"ee503b0d030092d92344c309ca6a9d82db5ad4cc3d0883751e53703f16c39411"} Nov 01 21:52:57.403055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.402157 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podStartSLOduration=1.260289552 podStartE2EDuration="5.402117869s" totalImagesPullingTime="4.141828317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:57.400867571 +0000 UTC m=+524.374770124" watchObservedRunningTime="2026-11-01 21:52:57.402117869 +0000 UTC m=+524.376020410" Nov 01 21:52:57.403055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.402837 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podStartSLOduration=1.537315392 podStartE2EDuration="5.402824308s" totalImagesPullingTime="3.865508916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:57.378202124 +0000 UTC m=+524.352104673" watchObservedRunningTime="2026-11-01 21:52:57.402824308 +0000 UTC m=+524.376726848" Nov 01 21:52:57.678492 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.675443 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:52:57.679284 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.679250 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:52:57.679502 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.679475 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:52:57.715222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:57.715185 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:52:58.682056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:58.678555 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:52:58.682056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:58.678654 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:52:58.722187 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:58.722147 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:52:58.722463 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:58.722433 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:52:59.682057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:59.678424 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:52:59.682057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:59.678487 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:52:59.716473 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:59.716435 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:52:59.716761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:52:59.716730 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:00.678455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:00.678415 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:53:00.678757 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:00.678727 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:53:00.715444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:00.715407 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:53:00.716406 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:00.716001 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:01.678485 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:01.678450 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:53:01.678779 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:01.678749 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:53:01.716442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:01.716403 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:53:01.717006 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:01.716975 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:02.317050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.316322 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:53:02.596099 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.594085 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5dd86d6db4-k28xn"] Nov 01 21:53:02.640479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.640426 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5dd86d6db4-k28xn"] Nov 01 21:53:02.640668 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.640581 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.697053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.696478 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:53:02.697053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.696563 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:53:02.724833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.724791 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:53:02.725249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.724849 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:02.742401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.742366 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwwv\" (UniqueName: \"kubernetes.io/projected/f5d35462-db7a-4cd7-bd40-427a4be8d921-kube-api-access-5zwwv\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.742552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.742440 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d35462-db7a-4cd7-bd40-427a4be8d921-metrics-certs\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.843280 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.843246 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d35462-db7a-4cd7-bd40-427a4be8d921-metrics-certs\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.843455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.843362 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwwv\" (UniqueName: \"kubernetes.io/projected/f5d35462-db7a-4cd7-bd40-427a4be8d921-kube-api-access-5zwwv\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.846303 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.846238 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d35462-db7a-4cd7-bd40-427a4be8d921-metrics-certs\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.855994 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.855971 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwwv\" (UniqueName: \"kubernetes.io/projected/f5d35462-db7a-4cd7-bd40-427a4be8d921-kube-api-access-5zwwv\") pod \"ai-gateway-controller-5dd86d6db4-k28xn\" (UID: \"f5d35462-db7a-4cd7-bd40-427a4be8d921\") " pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:02.953960 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:02.953912 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:03.263198 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.263173 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5dd86d6db4-k28xn"] Nov 01 21:53:03.266500 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:03.266467 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d35462_db7a_4cd7_bd40_427a4be8d921.slice/crio-0e850bd650c3f76d16844c735c2f1b13ff0fac0ce5328b50036b4cc2a3067b30 WatchSource:0}: Error finding container 0e850bd650c3f76d16844c735c2f1b13ff0fac0ce5328b50036b4cc2a3067b30: Status 404 returned error can't find the container with id 0e850bd650c3f76d16844c735c2f1b13ff0fac0ce5328b50036b4cc2a3067b30 Nov 01 21:53:03.394652 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.394617 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" event={"ID":"f5d35462-db7a-4cd7-bd40-427a4be8d921","Type":"ContainerStarted","Data":"0e850bd650c3f76d16844c735c2f1b13ff0fac0ce5328b50036b4cc2a3067b30"} Nov 01 21:53:03.675500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.675420 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vktwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 21:53:03.675500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.675476 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 21:53:03.714822 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.714778 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:53:03.714996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:03.714839 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:04.402071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:04.402018 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" event={"ID":"f5d35462-db7a-4cd7-bd40-427a4be8d921","Type":"ContainerStarted","Data":"0da0a7795e0b093ba2c9cd289798c7369c17f32cc9329c5d4ae929ae3a6f1ff1"} Nov 01 21:53:04.402553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:04.402186 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:04.421598 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:04.421549 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" podStartSLOduration=2.011261229 podStartE2EDuration="2.421533655s" totalImagesPullingTime="410.272426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:04.420359316 +0000 UTC m=+531.394261856" watchObservedRunningTime="2026-11-01 21:53:04.421533655 +0000 UTC m=+531.395436194" Nov 01 21:53:04.715639 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:04.715541 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vwrt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Nov 01 21:53:04.715639 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:04.715589 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Nov 01 21:53:05.297994 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.297957 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:53:05.298424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.298401 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:53:05.299422 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.299393 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 21:53:05.719136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.719065 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:53:05.719571 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.719325 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:53:05.719949 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:05.719934 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 21:53:07.355661 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.355633 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-tfddh" Nov 01 21:53:07.356077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.355795 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dxpk7" Nov 01 21:53:07.400981 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.400939 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8"] Nov 01 21:53:07.405392 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.405358 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.425859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.425829 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8"] Nov 01 21:53:07.589646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589598 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.589646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589647 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.589913 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589739 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.589913 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.590061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589922 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.590061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.589986 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.590177 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.590093 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.590236 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.590189 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.590236 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.590213 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd72\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-kube-api-access-nnd72\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691203 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691241 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd72\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-kube-api-access-nnd72\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691263 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691282 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691310 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691341 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691401 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691450 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691473 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691851 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691716 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691913 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691942 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.691996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.691941 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.692097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.692008 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.693725 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.693702 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.693982 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.693962 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.703773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.703748 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.703867 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.703814 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd72\" (UniqueName: \"kubernetes.io/projected/d2d23e65-78fd-4b1d-ae55-0e12d154c846-kube-api-access-nnd72\") pod \"maas-default-gateway-openshift-default-5bc884dd87-hcdq8\" (UID: \"d2d23e65-78fd-4b1d-ae55-0e12d154c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:07.725930 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:07.725904 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:08.004732 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.004707 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8"] Nov 01 21:53:08.007348 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:08.007318 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d23e65_78fd_4b1d_ae55_0e12d154c846.slice/crio-a681696964d2a521523c9fc1eef91df98ac5a2275b0418da3dc79d0011babede WatchSource:0}: Error finding container a681696964d2a521523c9fc1eef91df98ac5a2275b0418da3dc79d0011babede: Status 404 returned error can't find the container with id a681696964d2a521523c9fc1eef91df98ac5a2275b0418da3dc79d0011babede Nov 01 21:53:08.010425 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.010391 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:08.010523 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.010472 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:08.010523 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.010515 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:08.439346 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.438979 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" event={"ID":"d2d23e65-78fd-4b1d-ae55-0e12d154c846","Type":"ContainerStarted","Data":"17b6e73bb18a8ae6bffdca16927280d644ddf153953708b801217da00621b676"} Nov 01 21:53:08.439346 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.439083 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" event={"ID":"d2d23e65-78fd-4b1d-ae55-0e12d154c846","Type":"ContainerStarted","Data":"a681696964d2a521523c9fc1eef91df98ac5a2275b0418da3dc79d0011babede"} Nov 01 21:53:08.470050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.469970 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podStartSLOduration=1.469955243 podStartE2EDuration="1.469955243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:08.467792488 +0000 UTC m=+535.441695027" watchObservedRunningTime="2026-11-01 21:53:08.469955243 +0000 UTC m=+535.443857782" Nov 01 21:53:08.727099 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:08.726990 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:09.727915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:09.727878 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:09.728300 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:09.727935 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:10.726520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:10.726489 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:10.726668 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:10.726545 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:11.726607 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:11.726574 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:11.726996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:11.726622 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:12.726720 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:12.726685 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:12.727170 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:12.726753 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:13.726507 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:13.726467 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:13.726692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:13.726538 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:14.726870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:14.726829 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 21:53:14.727287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:14.726890 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 21:53:15.410733 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.410703 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5dd86d6db4-k28xn" Nov 01 21:53:15.458964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.458665 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:53:15.458964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.458905 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" podUID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" containerName="manager" containerID="cri-o://6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a" gracePeriod=30 Nov 01 21:53:15.730800 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.730764 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:53:15.763146 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.763110 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\" (UniqueName: \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\") pod \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " Nov 01 21:53:15.763324 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.763202 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\") pod \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\" (UID: \"1636fe18-e78a-46e5-b2ae-00b780e26d7e\") " Nov 01 21:53:15.765882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.765852 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs" pod "1636fe18-e78a-46e5-b2ae-00b780e26d7e" (UID: "1636fe18-e78a-46e5-b2ae-00b780e26d7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:53:15.766019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.765859 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr" pod "1636fe18-e78a-46e5-b2ae-00b780e26d7e" (UID: "1636fe18-e78a-46e5-b2ae-00b780e26d7e"). InnerVolumeSpecName "kube-api-access-rwlbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:53:15.864058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.864008 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1636fe18-e78a-46e5-b2ae-00b780e26d7e-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:53:15.864058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:15.864055 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwlbr\" (UniqueName: \"kubernetes.io/projected/1636fe18-e78a-46e5-b2ae-00b780e26d7e-kube-api-access-rwlbr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:53:16.253238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.253207 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:16.253921 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.253898 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:16.254671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.254655 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" Nov 01 21:53:16.488355 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.488319 4395 generic.go:356] "Generic (PLEG): container finished" podID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" containerID="6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a" exitCode=0 Nov 01 21:53:16.488493 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.488364 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" Nov 01 21:53:16.488493 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.488411 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" event={"ID":"1636fe18-e78a-46e5-b2ae-00b780e26d7e","Type":"ContainerDied","Data":"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a"} Nov 01 21:53:16.488493 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.488464 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-l9994" event={"ID":"1636fe18-e78a-46e5-b2ae-00b780e26d7e","Type":"ContainerDied","Data":"4a71ab5353fce6e37685092c1140e126d33ef2fa8a7994f6a67aea8cc03fd636"} Nov 01 21:53:16.488648 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.488495 4395 scope.go:122] "RemoveContainer" containerID="6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a" Nov 01 21:53:16.497883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.497862 4395 scope.go:122] "RemoveContainer" containerID="6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a" Nov 01 21:53:16.498193 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:53:16.498176 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a\": container with ID starting with 6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a not found: ID does not exist" containerID="6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a" Nov 01 21:53:16.498237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.498203 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a"} err="failed to get container status \"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a\": rpc error: code = NotFound desc = could not find container \"6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a\": container with ID starting with 6e28305627b458a9cfe2cc8df9aa5bca918b87f2107be352c5674e362f1f617a not found: ID does not exist" Nov 01 21:53:16.511938 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.511893 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:53:16.514291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:16.514273 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-l9994"] Nov 01 21:53:17.162963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:17.162927 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" path="/var/lib/kubelet/pods/1636fe18-e78a-46e5-b2ae-00b780e26d7e/volumes" Nov 01 21:53:22.300001 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:53:22.299959 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:53:22.428856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.428827 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 21:53:22.429267 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.429253 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" containerName="manager" Nov 01 21:53:22.429316 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.429270 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" containerName="manager" Nov 01 21:53:22.429351 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.429329 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1636fe18-e78a-46e5-b2ae-00b780e26d7e" containerName="manager" Nov 01 21:53:22.436575 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.436554 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.449167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.449140 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 21:53:22.515839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.515802 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.515977 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.515911 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p9q\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.515977 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.515957 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516165 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516174 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516314 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516299 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516370 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516327 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516417 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516378 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.516460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.516444 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617295 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617233 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617295 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617261 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617295 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617289 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617307 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617399 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617454 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p9q\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617495 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617537 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617576 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617750 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.617856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617780 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.618147 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617913 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.618147 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.617984 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.618258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.618227 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.619642 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.619614 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.619881 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.619862 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.629696 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.629671 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.629824 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.629679 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p9q\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c8vgk\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:22.755980 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:22.755954 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:23.033977 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.033947 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 21:53:23.036682 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:23.036654 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9688198f_1431_474e_9ac7_92266000c8c4.slice/crio-bf67789fb7a64b6dd8fe70158f8bd4615fa35d2f5e3c4ede6d81bf0935914b93 WatchSource:0}: Error finding container bf67789fb7a64b6dd8fe70158f8bd4615fa35d2f5e3c4ede6d81bf0935914b93: Status 404 returned error can't find the container with id bf67789fb7a64b6dd8fe70158f8bd4615fa35d2f5e3c4ede6d81bf0935914b93 Nov 01 21:53:23.038968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.038935 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:23.039089 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.039012 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:23.039089 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.039077 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:23.522416 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.522373 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" event={"ID":"9688198f-1431-474e-9ac7-92266000c8c4","Type":"ContainerStarted","Data":"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3"} Nov 01 21:53:23.522960 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.522438 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" event={"ID":"9688198f-1431-474e-9ac7-92266000c8c4","Type":"ContainerStarted","Data":"bf67789fb7a64b6dd8fe70158f8bd4615fa35d2f5e3c4ede6d81bf0935914b93"} Nov 01 21:53:23.550581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.550534 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podStartSLOduration=1.550521086 podStartE2EDuration="1.550521086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:23.547086559 +0000 UTC m=+550.520989098" watchObservedRunningTime="2026-11-01 21:53:23.550521086 +0000 UTC m=+550.524423673" Nov 01 21:53:23.756550 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:23.756513 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:24.757883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:24.757846 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:24.758381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:24.757918 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:25.757049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:25.756997 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:25.757232 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:25.757074 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:26.756880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:26.756845 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:26.757287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:26.756912 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:27.757098 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:27.757059 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:27.757508 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:27.757130 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:28.756814 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:28.756783 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:28.757070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:28.757042 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:29.756525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:29.756486 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 21:53:29.756921 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:29.756570 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 21:53:31.207955 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:31.207928 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:31.208421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:31.208341 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:31.209138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:31.209117 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 21:53:47.342090 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.342059 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 21:53:47.345407 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.345393 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.352498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.352475 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0def36-openshift-default-dockercfg-wfkj8\"" Nov 01 21:53:47.358548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.358527 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 21:53:47.454198 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454158 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454236 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454276 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454316 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454371 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454440 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454490 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454524 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.454742 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.454583 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmfg\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555473 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555674 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555539 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555674 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555571 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555674 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555589 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555674 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555634 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmfg\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555695 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555731 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555770 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555800 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.555882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555854 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.556202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555906 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.556202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.555959 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.556309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.556257 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.556500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.556484 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.557940 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.557921 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.558286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.558267 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.567845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.567825 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.567964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.567866 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmfg\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.611006 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.610932 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 21:53:47.616572 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.615791 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.637463 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.637438 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f033b1-openshift-default-dockercfg-svg6v\"" Nov 01 21:53:47.639996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.639969 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 21:53:47.660691 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.660664 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:47.757220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757186 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757295 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757359 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757326 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757381 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757407 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757434 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757428 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75tx\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757449 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757599 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757540 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.757633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.757604 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.809110 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.809077 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 21:53:47.813096 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.813077 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.824981 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.824957 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 21:53:47.850299 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.850276 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-53b784-openshift-default-dockercfg-m5k4p\"" Nov 01 21:53:47.858701 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858681 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858727 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858760 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858809 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858842 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75tx\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858876 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858913 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.858988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.858972 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859022 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859121 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859275 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859384 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859721 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859700 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.859892 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.859864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.861230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.861193 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.861408 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.861392 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.877919 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.877891 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.878964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.878943 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75tx\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.930167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.930144 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:47.951918 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.951895 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 21:53:47.953443 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:47.953416 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7185eb_26b9_4542_9552_17cd22207048.slice/crio-fa8756064fa286d77c0628040d2bfe1860fa41252295a5df267ff9b5b1f93aaf WatchSource:0}: Error finding container fa8756064fa286d77c0628040d2bfe1860fa41252295a5df267ff9b5b1f93aaf: Status 404 returned error can't find the container with id fa8756064fa286d77c0628040d2bfe1860fa41252295a5df267ff9b5b1f93aaf Nov 01 21:53:47.955882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.955852 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:47.955947 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.955917 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:47.955984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.955945 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:47.960237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960212 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960316 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960255 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960375 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960357 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960480 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960458 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960586 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960568 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960621 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960603 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6tk\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960740 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960661 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960740 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960700 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:47.960740 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:47.960741 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081609 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081693 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081755 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081782 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6tk\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081811 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081848 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081893 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081940 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.081963 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.083891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.083235 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.087183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.084408 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.087183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.084655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.087183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.085206 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.093923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.087878 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.099499 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.099435 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.102401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.102319 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6tk\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.106086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.103125 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.106086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.105858 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.131159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.130929 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:48.255660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.253198 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 21:53:48.255660 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:48.254388 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4399b1_1c58_4e95_bfc3_3cf1dfdc024f.slice/crio-6eb3415f5b71d6a847f53e2a2fa6007018963296e6c9a7fff0a464e1e98662d9 WatchSource:0}: Error finding container 6eb3415f5b71d6a847f53e2a2fa6007018963296e6c9a7fff0a464e1e98662d9: Status 404 returned error can't find the container with id 6eb3415f5b71d6a847f53e2a2fa6007018963296e6c9a7fff0a464e1e98662d9 Nov 01 21:53:48.259734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.257966 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.259734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.258065 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.259734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.258107 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.495739 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.495707 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 21:53:48.497784 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:48.497739 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059a38f7_1f72_418d_9c8a_fa3e86a8674c.slice/crio-b57f11d3e247d00e3054d3c698b29b4b6731bf2640f339e680da253bd129bc9b WatchSource:0}: Error finding container b57f11d3e247d00e3054d3c698b29b4b6731bf2640f339e680da253bd129bc9b: Status 404 returned error can't find the container with id b57f11d3e247d00e3054d3c698b29b4b6731bf2640f339e680da253bd129bc9b Nov 01 21:53:48.500862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.500820 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.501200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.500908 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.501200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.500958 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:53:48.638061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.638010 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerStarted","Data":"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795"} Nov 01 21:53:48.638338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.638309 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerStarted","Data":"b57f11d3e247d00e3054d3c698b29b4b6731bf2640f339e680da253bd129bc9b"} Nov 01 21:53:48.640877 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.640798 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerStarted","Data":"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012"} Nov 01 21:53:48.640877 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.640844 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerStarted","Data":"6eb3415f5b71d6a847f53e2a2fa6007018963296e6c9a7fff0a464e1e98662d9"} Nov 01 21:53:48.643729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.643703 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerStarted","Data":"c51d04faf14256c305a28f0f1ee8fdee90732696b88977e6a21129bd95d7bd51"} Nov 01 21:53:48.643860 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.643750 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerStarted","Data":"fa8756064fa286d77c0628040d2bfe1860fa41252295a5df267ff9b5b1f93aaf"} Nov 01 21:53:48.660965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.660937 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:48.663908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.663864 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podStartSLOduration=1.663852086 podStartE2EDuration="1.663852086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:48.662009252 +0000 UTC m=+575.635911803" watchObservedRunningTime="2026-11-01 21:53:48.663852086 +0000 UTC m=+575.637754625" Nov 01 21:53:48.687809 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.687722 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podStartSLOduration=1.6877066379999999 podStartE2EDuration="1.687706638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:48.6845811 +0000 UTC m=+575.658483640" watchObservedRunningTime="2026-11-01 21:53:48.687706638 +0000 UTC m=+575.661609177" Nov 01 21:53:48.716913 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.710517 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podStartSLOduration=1.710501863 podStartE2EDuration="1.710501863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:48.707085615 +0000 UTC m=+575.680988155" watchObservedRunningTime="2026-11-01 21:53:48.710501863 +0000 UTC m=+575.684404404" Nov 01 21:53:48.931005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:48.930971 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:49.131977 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.131949 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:49.137740 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.137713 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:49.648363 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.648330 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:49.649139 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.649121 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:53:49.666754 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.666732 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:49.936322 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:49.936241 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:50.652692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:50.652657 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:50.653132 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:50.652743 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:50.653258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:50.653239 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:53:50.653388 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:50.653369 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:53:52.150736 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.150700 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 21:53:52.154583 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.154565 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.163390 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.163365 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1f3a6b\"" Nov 01 21:53:52.165899 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.165877 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 21:53:52.223141 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.223101 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.223278 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.223260 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pjj\" (UniqueName: \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.330063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.329652 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.330063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.329805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pjj\" (UniqueName: \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.334059 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.331137 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.368420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.368391 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pjj\" (UniqueName: \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\") pod \"payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.391091 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:53:52.391060 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:53:52.474004 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.473919 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:52.756876 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.756854 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 21:53:52.759092 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:52.759067 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272d608c_9bc8_4c68_a86a_5ef2ef123625.slice/crio-2d6ea11bd96db22494f3c65918688be816b7ffef258decda8e6f722195f4d816 WatchSource:0}: Error finding container 2d6ea11bd96db22494f3c65918688be816b7ffef258decda8e6f722195f4d816: Status 404 returned error can't find the container with id 2d6ea11bd96db22494f3c65918688be816b7ffef258decda8e6f722195f4d816 Nov 01 21:53:52.844347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.844322 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 21:53:52.847904 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.847880 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:52.855439 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.855420 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-fb9597\"" Nov 01 21:53:52.863657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.862701 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 21:53:52.936294 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.936269 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:52.936405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:52.936330 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9s2\" (UniqueName: \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.039798 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.039700 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.039798 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.039791 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9s2\" (UniqueName: \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.040650 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.040624 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.053327 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.053299 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9s2\" (UniqueName: \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\") pod \"payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.160966 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.160942 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.440744 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.440711 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 21:53:53.443120 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:53.443093 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d34f11_33b0_4247_8fbc_ebce0b2a13d7.slice/crio-1581564ff501de85ece172fa1de64946cf671178feb80c328004b62601b8b294 WatchSource:0}: Error finding container 1581564ff501de85ece172fa1de64946cf671178feb80c328004b62601b8b294: Status 404 returned error can't find the container with id 1581564ff501de85ece172fa1de64946cf671178feb80c328004b62601b8b294 Nov 01 21:53:53.673169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.671999 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" event={"ID":"56d34f11-33b0-4247-8fbc-ebce0b2a13d7","Type":"ContainerStarted","Data":"f5da4ed42ecd616cff0c36af64d07f517d4ccdd9ac728cc443e045b3c2a67860"} Nov 01 21:53:53.673169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.672144 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:53:53.673169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.672171 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" event={"ID":"56d34f11-33b0-4247-8fbc-ebce0b2a13d7","Type":"ContainerStarted","Data":"1581564ff501de85ece172fa1de64946cf671178feb80c328004b62601b8b294"} Nov 01 21:53:53.673979 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.673949 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" event={"ID":"272d608c-9bc8-4c68-a86a-5ef2ef123625","Type":"ContainerStarted","Data":"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856"} Nov 01 21:53:53.674169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.673996 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" event={"ID":"272d608c-9bc8-4c68-a86a-5ef2ef123625","Type":"ContainerStarted","Data":"2d6ea11bd96db22494f3c65918688be816b7ffef258decda8e6f722195f4d816"} Nov 01 21:53:53.674169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.674041 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:53:53.692996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.692923 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" podStartSLOduration=1.692910937 podStartE2EDuration="1.692910937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:53.690448923 +0000 UTC m=+580.664351463" watchObservedRunningTime="2026-11-01 21:53:53.692910937 +0000 UTC m=+580.666813514" Nov 01 21:53:53.712181 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.712141 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" podStartSLOduration=1.712125346 podStartE2EDuration="1.712125346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:53.710951153 +0000 UTC m=+580.684853693" watchObservedRunningTime="2026-11-01 21:53:53.712125346 +0000 UTC m=+580.686027885" Nov 01 21:53:53.768923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.768896 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 21:53:53.772509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.772489 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.783502 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.781682 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f033b1\"" Nov 01 21:53:53.785176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.785153 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 21:53:53.792133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.792114 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 21:53:53.795846 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.795828 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.805838 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.805815 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f033b1-dockercfg-zvtnf\"" Nov 01 21:53:53.809138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.809116 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 21:53:53.847217 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.847193 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lwm\" (UniqueName: \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.847329 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.847225 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.847417 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.847398 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655q9\" (UniqueName: \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.847467 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.847455 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.951736 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.951641 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lwm\" (UniqueName: \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.951969 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.951951 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.952250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.952232 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655q9\" (UniqueName: \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.952401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.952387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.953350 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.953324 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:53.954669 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.954167 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.969995 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.969948 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655q9\" (UniqueName: \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\") pod \"payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:53.970417 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:53.970396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lwm\" (UniqueName: \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\") pod \"payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:54.099698 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.099664 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:54.118676 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.118649 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:54.405655 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.405626 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 21:53:54.410044 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:54.408287 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5261cb1c_37ea_43a6_8edd_55e1c3f624e2.slice/crio-4fa672436068c9402f91a7955c3475f9a2240e5e9724244dff905d53df80042a WatchSource:0}: Error finding container 4fa672436068c9402f91a7955c3475f9a2240e5e9724244dff905d53df80042a: Status 404 returned error can't find the container with id 4fa672436068c9402f91a7955c3475f9a2240e5e9724244dff905d53df80042a Nov 01 21:53:54.420060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.420001 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 21:53:54.423074 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:54.422256 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6238bac1_202d_49f0_9c48_3bc87b9158ae.slice/crio-b1213e4bf2aaafd3bbe934422c2e8eb3dc70adcd654ed211a4ab771d4d8ec996 WatchSource:0}: Error finding container b1213e4bf2aaafd3bbe934422c2e8eb3dc70adcd654ed211a4ab771d4d8ec996: Status 404 returned error can't find the container with id b1213e4bf2aaafd3bbe934422c2e8eb3dc70adcd654ed211a4ab771d4d8ec996 Nov 01 21:53:54.578551 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.578520 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 21:53:54.582512 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.582492 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.594992 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.594965 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0def36\"" Nov 01 21:53:54.596978 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.596949 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 21:53:54.599441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.599417 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 21:53:54.603400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.603379 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.614402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.614378 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 21:53:54.615259 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.615239 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0def36-dockercfg-zz8w2\"" Nov 01 21:53:54.665428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.665401 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsk2h\" (UniqueName: \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.665527 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.665442 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.665527 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.665476 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.665649 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.665602 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bjz\" (UniqueName: \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.680735 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.680708 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" event={"ID":"6238bac1-202d-49f0-9c48-3bc87b9158ae","Type":"ContainerStarted","Data":"935d7e07f8c0a6711b3e020b1be296c0ba9c65a4ee6cec0f1773a7107c73b7e2"} Nov 01 21:53:54.680826 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.680750 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" event={"ID":"6238bac1-202d-49f0-9c48-3bc87b9158ae","Type":"ContainerStarted","Data":"b1213e4bf2aaafd3bbe934422c2e8eb3dc70adcd654ed211a4ab771d4d8ec996"} Nov 01 21:53:54.680913 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.680897 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:53:54.682616 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.682592 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" event={"ID":"5261cb1c-37ea-43a6-8edd-55e1c3f624e2","Type":"ContainerStarted","Data":"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e"} Nov 01 21:53:54.682710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.682636 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" event={"ID":"5261cb1c-37ea-43a6-8edd-55e1c3f624e2","Type":"ContainerStarted","Data":"4fa672436068c9402f91a7955c3475f9a2240e5e9724244dff905d53df80042a"} Nov 01 21:53:54.682862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.682847 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:53:54.723424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.723328 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" podStartSLOduration=1.72331517 podStartE2EDuration="1.72331517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:54.702292994 +0000 UTC m=+581.676195534" watchObservedRunningTime="2026-11-01 21:53:54.72331517 +0000 UTC m=+581.697217702" Nov 01 21:53:54.725846 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.725810 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" podStartSLOduration=1.7257991449999999 podStartE2EDuration="1.725799145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:54.722749545 +0000 UTC m=+581.696652096" watchObservedRunningTime="2026-11-01 21:53:54.725799145 +0000 UTC m=+581.699701683" Nov 01 21:53:54.766638 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.766612 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bjz\" (UniqueName: \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.766769 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.766750 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsk2h\" (UniqueName: \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.767231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.766897 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.767231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.766997 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.767600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.767581 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.767653 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.767634 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.779299 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.779278 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsk2h\" (UniqueName: \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\") pod \"payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.779469 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.779451 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bjz\" (UniqueName: \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\") pod \"payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:54.897266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.897232 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:54.927465 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:54.927436 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:55.196260 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.196233 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 21:53:55.197399 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:55.197376 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04819ee_0d0d_4929_b94e_15f87ffc1228.slice/crio-3ac5739fb196b88e5f7d28d61a8495e7d0f10df5767279bbf060f57f4bb79e1c WatchSource:0}: Error finding container 3ac5739fb196b88e5f7d28d61a8495e7d0f10df5767279bbf060f57f4bb79e1c: Status 404 returned error can't find the container with id 3ac5739fb196b88e5f7d28d61a8495e7d0f10df5767279bbf060f57f4bb79e1c Nov 01 21:53:55.223211 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.223193 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 21:53:55.231111 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:55.231089 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50de95f8_e75d_4cb4_be29_ba73f9135388.slice/crio-d5f724df42f9b722e89485937546c7f32c5c3553d1e42a562c781e94feb9df91 WatchSource:0}: Error finding container d5f724df42f9b722e89485937546c7f32c5c3553d1e42a562c781e94feb9df91: Status 404 returned error can't find the container with id d5f724df42f9b722e89485937546c7f32c5c3553d1e42a562c781e94feb9df91 Nov 01 21:53:55.522987 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.522958 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 21:53:55.526790 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.526767 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.536264 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.536236 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 21:53:55.571600 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.571548 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ff3e70\"" Nov 01 21:53:55.576067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.576022 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.576249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.576230 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbcz\" (UniqueName: \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.625424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.625388 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ff3e70-dockercfg-4qxmz\"" Nov 01 21:53:55.679786 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.679731 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbcz\" (UniqueName: \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.679975 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.679839 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.680840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.680812 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.699732 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.699703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbcz\" (UniqueName: \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\") pod \"payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:55.700403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.700375 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" event={"ID":"c04819ee-0d0d-4929-b94e-15f87ffc1228","Type":"ContainerStarted","Data":"ec83eab834fc5ee337c32b6e5a96de39c2008224c6ef25b0a8b9553185561202"} Nov 01 21:53:55.700605 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.700587 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:53:55.700744 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.700726 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" event={"ID":"c04819ee-0d0d-4929-b94e-15f87ffc1228","Type":"ContainerStarted","Data":"3ac5739fb196b88e5f7d28d61a8495e7d0f10df5767279bbf060f57f4bb79e1c"} Nov 01 21:53:55.702392 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.702360 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" event={"ID":"50de95f8-e75d-4cb4-be29-ba73f9135388","Type":"ContainerStarted","Data":"13dec8004fee61986a613c5ddab33d216bf61e2c051c4b98a1db8ccd5d33c15f"} Nov 01 21:53:55.702490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.702407 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" event={"ID":"50de95f8-e75d-4cb4-be29-ba73f9135388","Type":"ContainerStarted","Data":"d5f724df42f9b722e89485937546c7f32c5c3553d1e42a562c781e94feb9df91"} Nov 01 21:53:55.702555 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.702485 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:53:55.721168 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.721131 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" podStartSLOduration=1.721118562 podStartE2EDuration="1.721118562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:55.717844401 +0000 UTC m=+582.691746940" watchObservedRunningTime="2026-11-01 21:53:55.721118562 +0000 UTC m=+582.695021101" Nov 01 21:53:55.735939 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.735900 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" podStartSLOduration=1.735889109 podStartE2EDuration="1.735889109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:55.735755638 +0000 UTC m=+582.709658179" watchObservedRunningTime="2026-11-01 21:53:55.735889109 +0000 UTC m=+582.709791689" Nov 01 21:53:55.844835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:55.844758 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:56.125401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.125368 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 21:53:56.127147 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:56.127110 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b827fa_0b84_4c05_812b_50ddf3e8d181.slice/crio-224fbbab2322e97b84571451f4237a5c94f773f1c09fbdcacda6c419d3f36b0d WatchSource:0}: Error finding container 224fbbab2322e97b84571451f4237a5c94f773f1c09fbdcacda6c419d3f36b0d: Status 404 returned error can't find the container with id 224fbbab2322e97b84571451f4237a5c94f773f1c09fbdcacda6c419d3f36b0d Nov 01 21:53:56.680247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.680216 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 21:53:56.684149 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.684123 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.687971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.687935 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f033b1-serving-cert\"" Nov 01 21:53:56.689249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.689226 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zltrj\"" Nov 01 21:53:56.689388 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.689228 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 21:53:56.698510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.692949 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.698510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.693064 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.698510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.693102 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrnh\" (UniqueName: \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.698510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.693171 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.698510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.697951 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 21:53:56.711099 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.711070 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" event={"ID":"b8b827fa-0b84-4c05-812b-50ddf3e8d181","Type":"ContainerStarted","Data":"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2"} Nov 01 21:53:56.711212 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.711120 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" event={"ID":"b8b827fa-0b84-4c05-812b-50ddf3e8d181","Type":"ContainerStarted","Data":"224fbbab2322e97b84571451f4237a5c94f773f1c09fbdcacda6c419d3f36b0d"} Nov 01 21:53:56.711518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.711501 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:53:56.731965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.731916 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" podStartSLOduration=1.7319061740000001 podStartE2EDuration="1.731906174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:56.729326962 +0000 UTC m=+583.703229505" watchObservedRunningTime="2026-11-01 21:53:56.731906174 +0000 UTC m=+583.705808713" Nov 01 21:53:56.794133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.794107 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.794232 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.794150 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrnh\" (UniqueName: \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.794295 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.794234 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.794295 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:53:56.794239 4395 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-serving-cert: secret "maas-api-e2e-worker-w2-f033b1-serving-cert" not found Nov 01 21:53:56.794434 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:53:56.794411 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls podName:20b9c9f5-f9b2-403f-90b2-f484b4e9a735 nodeName:}" failed. No retries permitted until 2026-11-01 21:53:57.294383753 +0000 UTC m=+584.268286288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls") pod "maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" (UID: "20b9c9f5-f9b2-403f-90b2-f484b4e9a735") : secret "maas-api-e2e-worker-w2-f033b1-serving-cert" not found Nov 01 21:53:56.794577 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.794518 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.795174 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.795157 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.796775 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.796756 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:56.807661 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:56.807643 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrnh\" (UniqueName: \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:57.305863 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.305829 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:57.308519 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.308492 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:57.311224 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.311202 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:57.626437 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.626163 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 21:53:57.629134 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:57.629103 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b9c9f5_f9b2_403f_90b2_f484b4e9a735.slice/crio-0d27022c20e492529ab3e1fa3ee919becea92b9bd1b5ddbfc19bd8c184d7892b WatchSource:0}: Error finding container 0d27022c20e492529ab3e1fa3ee919becea92b9bd1b5ddbfc19bd8c184d7892b: Status 404 returned error can't find the container with id 0d27022c20e492529ab3e1fa3ee919becea92b9bd1b5ddbfc19bd8c184d7892b Nov 01 21:53:57.638806 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.638767 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 21:53:57.656525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.656449 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 21:53:57.657402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.656751 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.662715 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.662697 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-53b784-serving-cert\"" Nov 01 21:53:57.717597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.717571 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.718018 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.717628 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2rb\" (UniqueName: \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.718018 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.717761 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.718018 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.717796 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.718220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.718084 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" event={"ID":"20b9c9f5-f9b2-403f-90b2-f484b4e9a735","Type":"ContainerStarted","Data":"0d27022c20e492529ab3e1fa3ee919becea92b9bd1b5ddbfc19bd8c184d7892b"} Nov 01 21:53:57.819441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.819408 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.819629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.819467 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2rb\" (UniqueName: \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.819629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.819514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.819629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.819539 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.820318 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.820292 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.821983 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.821962 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.822099 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.821991 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.831403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.831380 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2rb\" (UniqueName: \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\") pod \"maas-api-e2e-worker-w1-53b784-589599c67-2jt4l\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:57.972185 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:57.972110 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:58.248890 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:58.248863 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 21:53:58.250828 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:53:58.250798 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8914d34d_6ae3_4767_b974_e3674b1212eb.slice/crio-29661c95f3d9583835ea9afa5b1cf48682d855008c1b604cc8a4d26fe4c6aa6d WatchSource:0}: Error finding container 29661c95f3d9583835ea9afa5b1cf48682d855008c1b604cc8a4d26fe4c6aa6d: Status 404 returned error can't find the container with id 29661c95f3d9583835ea9afa5b1cf48682d855008c1b604cc8a4d26fe4c6aa6d Nov 01 21:53:58.725675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:58.725649 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" event={"ID":"8914d34d-6ae3-4767-b974-e3674b1212eb","Type":"ContainerStarted","Data":"29661c95f3d9583835ea9afa5b1cf48682d855008c1b604cc8a4d26fe4c6aa6d"} Nov 01 21:53:58.727498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:58.727470 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" event={"ID":"20b9c9f5-f9b2-403f-90b2-f484b4e9a735","Type":"ContainerStarted","Data":"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08"} Nov 01 21:53:58.727606 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:58.727589 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:53:58.748610 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:58.748559 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" podStartSLOduration=2.344043225 podStartE2EDuration="2.748546393s" totalImagesPullingTime="404.503168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:58.746342544 +0000 UTC m=+585.720245084" watchObservedRunningTime="2026-11-01 21:53:58.748546393 +0000 UTC m=+585.722448932" Nov 01 21:53:59.735132 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:59.735100 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" event={"ID":"8914d34d-6ae3-4767-b974-e3674b1212eb","Type":"ContainerStarted","Data":"93af37a4cfe9385718180c16dabc5d900d918a6c6c02aefd853ab0ca5d71698c"} Nov 01 21:53:59.735621 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:59.735199 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:53:59.758116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:53:59.758067 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" podStartSLOduration=2.3529653440000002 podStartE2EDuration="2.758053343s" totalImagesPullingTime="405.087999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:59.75582892 +0000 UTC m=+586.729731462" watchObservedRunningTime="2026-11-01 21:53:59.758053343 +0000 UTC m=+586.731955882" Nov 01 21:54:01.053648 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.053556 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 21:54:01.117261 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.104481 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.117818 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.117785 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-b0b7a3-serving-cert\"" Nov 01 21:54:01.118632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.118609 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 21:54:01.155813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.155791 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.155949 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.155844 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.155949 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.155928 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.156098 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.156000 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwdc\" (UniqueName: \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.257142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.257109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.257483 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.257468 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.257792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.257772 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.258133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.258111 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwdc\" (UniqueName: \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.259346 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.259322 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.260860 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.260831 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.264146 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.261534 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.271344 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.271321 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwdc\" (UniqueName: \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\") pod \"maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.426877 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.426784 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:01.772154 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:01.771007 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 21:54:02.807736 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:02.807703 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" event={"ID":"974aea98-ab66-46d1-803b-ef25ee14b91b","Type":"ContainerStarted","Data":"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326"} Nov 01 21:54:02.808347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:02.808323 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" event={"ID":"974aea98-ab66-46d1-803b-ef25ee14b91b","Type":"ContainerStarted","Data":"d6d182896dc7c943f6a0208cce4c4342cf2f6023d69d7447638d6b7a734259bd"} Nov 01 21:54:02.826054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:02.809313 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:02.834746 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:02.834693 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" podStartSLOduration=1.387444301 podStartE2EDuration="1.834673271s" totalImagesPullingTime="447.22897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:02.831381859 +0000 UTC m=+589.805284399" watchObservedRunningTime="2026-11-01 21:54:02.834673271 +0000 UTC m=+589.808575813" Nov 01 21:54:04.729283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:04.729252 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 21:54:04.736276 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:04.736250 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 21:54:04.764156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:04.764123 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 21:54:05.751287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:05.751251 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 21:54:05.753854 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:05.753831 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 21:54:05.756063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:05.755965 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 21:54:06.731807 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:06.731769 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 21:54:07.721268 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:07.721241 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 21:54:07.721729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:07.721310 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 21:54:09.862041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:09.861994 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 21:54:11.968188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.968096 4395 generic.go:356] "Generic (PLEG): container finished" podID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerID="654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795" exitCode=0 Nov 01 21:54:11.968775 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.968264 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerDied","Data":"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795"} Nov 01 21:54:11.969430 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.969408 4395 scope.go:122] "RemoveContainer" containerID="654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795" Nov 01 21:54:11.970792 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.970758 4395 generic.go:356] "Generic (PLEG): container finished" podID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerID="b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012" exitCode=0 Nov 01 21:54:11.970875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.970853 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerDied","Data":"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012"} Nov 01 21:54:11.971162 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.971130 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:11.971238 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.971220 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:11.971279 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.971242 4395 scope.go:122] "RemoveContainer" containerID="b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012" Nov 01 21:54:11.971309 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.971272 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:11.973970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.973939 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:11.974056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.974043 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:11.974116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:11.974093 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:12.992056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:12.990931 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerStarted","Data":"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275"} Nov 01 21:54:13.020051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:12.998269 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerStarted","Data":"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228"} Nov 01 21:54:13.138445 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.138408 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:54:13.140049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.139997 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:13.140193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.140105 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:13.931374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.931302 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 21:54:13.931622 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.931594 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 21:54:13.937131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:13.937100 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:54:14.133896 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:14.133862 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:14.134393 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:14.134367 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:14.930797 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:14.930752 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 21:54:14.931126 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:14.931087 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 21:54:15.132709 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:15.132659 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:15.132917 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:15.132735 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:15.931204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:15.931160 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 21:54:15.931722 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:15.931220 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 21:54:16.132514 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:16.132467 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:16.132705 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:16.132542 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:16.932054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:16.931503 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 21:54:16.932054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:16.931569 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 21:54:17.073837 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.073800 4395 generic.go:356] "Generic (PLEG): container finished" podID="ec7185eb-26b9-4542-9552-17cd22207048" containerID="c51d04faf14256c305a28f0f1ee8fdee90732696b88977e6a21129bd95d7bd51" exitCode=0 Nov 01 21:54:17.074008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.073875 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerDied","Data":"c51d04faf14256c305a28f0f1ee8fdee90732696b88977e6a21129bd95d7bd51"} Nov 01 21:54:17.075518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.074379 4395 scope.go:122] "RemoveContainer" containerID="c51d04faf14256c305a28f0f1ee8fdee90732696b88977e6a21129bd95d7bd51" Nov 01 21:54:17.076799 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.076765 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:17.084409 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.077420 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:17.084615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.084593 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:54:17.135996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.135565 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:17.135996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.135623 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:17.934433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.934390 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 21:54:17.935005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:17.934973 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 21:54:18.150191 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.150152 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 21:54:18.150477 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.150447 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 21:54:18.150879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.150855 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerStarted","Data":"a4f62600334ccaa65fd33e535e7fdd4090812ca0b4ae28500b9bedc2134ff876"} Nov 01 21:54:18.661400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.661358 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:54:18.662879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.662843 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:18.663071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.662910 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:18.958136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.958004 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:54:18.959111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.959079 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:54:18.960755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:18.960739 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 21:54:19.135652 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:19.135625 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:54:19.136072 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:19.136053 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:54:19.136723 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:19.136705 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 21:54:19.661397 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:19.661366 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:19.661560 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:19.661420 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:20.652433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:20.652401 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:54:20.662880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:20.662476 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:20.662880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:20.662534 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:21.661382 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:21.661347 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:21.661761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:21.661412 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:22.324137 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:54:22.324100 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:54:22.661452 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:22.661370 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:22.661452 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:22.661435 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:23.661563 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:23.661527 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 21:54:23.661946 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:23.661582 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 21:54:24.665101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:24.665066 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:54:24.666095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:24.666074 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 21:54:40.198608 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.198566 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 21:54:40.203207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.203184 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.214008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.213977 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 21:54:40.215544 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.215500 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"default-dockercfg-bqm9h\"" Nov 01 21:54:40.215685 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.215637 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"premium-w0-1f3a6b-kserve-self-signed-certs\"" Nov 01 21:54:40.215685 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.215684 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"kube-root-ca.crt\"" Nov 01 21:54:40.215817 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.215732 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"openshift-service-ca.crt\"" Nov 01 21:54:40.365052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.345928 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.365472 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.365418 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdp4\" (UniqueName: \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.365869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.365843 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.366138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.366116 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.366339 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.366315 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.366478 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.366456 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.467813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467726 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.467813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467793 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdp4\" (UniqueName: \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467823 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467847 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467880 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.467922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.468237 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.468373 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.468697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.468664 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.471078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.471019 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.471244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.471225 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.480484 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.480457 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdp4\" (UniqueName: \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\") pod \"premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.521430 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.521407 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:40.797426 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:40.797404 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 21:54:40.799222 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:54:40.799189 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff4e66d_2207_4b0d_bf2e_e9a5dc62b41e.slice/crio-4e7c8852a474356181c5f2082536e37107ac24e4c8c2b7f70057ee398b54eb7e WatchSource:0}: Error finding container 4e7c8852a474356181c5f2082536e37107ac24e4c8c2b7f70057ee398b54eb7e: Status 404 returned error can't find the container with id 4e7c8852a474356181c5f2082536e37107ac24e4c8c2b7f70057ee398b54eb7e Nov 01 21:54:41.383264 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:41.383229 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" event={"ID":"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e","Type":"ContainerStarted","Data":"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f"} Nov 01 21:54:41.383711 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:41.383275 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" event={"ID":"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e","Type":"ContainerStarted","Data":"4e7c8852a474356181c5f2082536e37107ac24e4c8c2b7f70057ee398b54eb7e"} Nov 01 21:54:41.402443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:41.402398 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podStartSLOduration=1.402381644 podStartE2EDuration="1.402381644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:41.401987967 +0000 UTC m=+628.375890521" watchObservedRunningTime="2026-11-01 21:54:41.402381644 +0000 UTC m=+628.376284184" Nov 01 21:54:41.523552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:41.523489 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:45.105506 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.105472 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 21:54:45.109476 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.109455 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.115965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.115925 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"default-dockercfg-prhv9\"" Nov 01 21:54:45.118834 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.118810 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"premium-w3-ff3e70-kserve-self-signed-certs\"" Nov 01 21:54:45.118984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.118848 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"openshift-service-ca.crt\"" Nov 01 21:54:45.119205 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.119191 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"kube-root-ca.crt\"" Nov 01 21:54:45.121547 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.121518 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 21:54:45.226878 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.226850 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzbz\" (UniqueName: \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.227007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.226883 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.227007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.226953 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.227007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.226996 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.227153 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.227122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.227153 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.227147 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.328640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328607 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzbz\" (UniqueName: \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.328842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328649 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.328842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328672 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.328842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328795 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.329005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328876 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.329005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.328896 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.329185 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.329164 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.329289 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.329268 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.329334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.329311 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.331047 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.331014 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.331341 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.331322 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.344367 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.344346 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzbz\" (UniqueName: \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\") pod \"premium-w3-ff3e70-kserve-6c6c494798-ss849\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.427239 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.427184 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:45.704162 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:45.704118 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 21:54:45.705900 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:54:45.705876 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8640e002_c647_4ac9_813e_d93f131dfcb2.slice/crio-2af3c14bdd8550eba9f02251e14529946f79b162cd28a6ff3b4f02dd9e6798a6 WatchSource:0}: Error finding container 2af3c14bdd8550eba9f02251e14529946f79b162cd28a6ff3b4f02dd9e6798a6: Status 404 returned error can't find the container with id 2af3c14bdd8550eba9f02251e14529946f79b162cd28a6ff3b4f02dd9e6798a6 Nov 01 21:54:46.416657 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:46.416628 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" event={"ID":"8640e002-c647-4ac9-813e-d93f131dfcb2","Type":"ContainerStarted","Data":"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9"} Nov 01 21:54:46.417051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:46.416666 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" event={"ID":"8640e002-c647-4ac9-813e-d93f131dfcb2","Type":"ContainerStarted","Data":"2af3c14bdd8550eba9f02251e14529946f79b162cd28a6ff3b4f02dd9e6798a6"} Nov 01 21:54:46.428003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:46.427981 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:46.439750 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:46.439710 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podStartSLOduration=1.439697263 podStartE2EDuration="1.439697263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:46.437425306 +0000 UTC m=+633.411327847" watchObservedRunningTime="2026-11-01 21:54:46.439697263 +0000 UTC m=+633.413599802" Nov 01 21:54:50.522305 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:50.522270 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:50.534846 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:50.534821 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:51.452003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:51.451977 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:54:52.304674 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:54:52.304631 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:54:55.427651 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:55.427619 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:55.440492 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:55.440468 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:55.473835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:55.473811 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:54:58.742019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.741989 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 21:54:58.751407 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.751381 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.755599 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.755572 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 21:54:58.756616 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.756562 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"kube-root-ca.crt\"" Nov 01 21:54:58.757122 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.757100 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"default-dockercfg-bpmhg\"" Nov 01 21:54:58.757298 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.757154 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"openshift-service-ca.crt\"" Nov 01 21:54:58.757412 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.757273 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"premium-w1-53b784-kserve-self-signed-certs\"" Nov 01 21:54:58.808334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808307 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.808481 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808347 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.808481 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808450 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ntg\" (UniqueName: \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.808589 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808519 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.808623 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808597 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.808656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.808637 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913202 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913162 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913491 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913463 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913628 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913526 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913628 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913620 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913745 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913666 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.913806 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.913757 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ntg\" (UniqueName: \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.915207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.915179 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.915485 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.915459 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.915708 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.915683 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.927110 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.927055 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.936653 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.936626 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ntg\" (UniqueName: \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:58.937667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:58.937633 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\") pod \"premium-w1-53b784-kserve-98b787848-tzpc6\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:59.067251 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:59.067219 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:54:59.345546 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:59.345525 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 21:54:59.485821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:59.485794 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" event={"ID":"5fe2733d-471f-43d6-b75e-bf652ca28621","Type":"ContainerStarted","Data":"9780100779d3f7cad1e6565a6d94eec35e39aba551e0aa19a5cf334266145ee6"} Nov 01 21:54:59.485963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:59.485832 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" event={"ID":"5fe2733d-471f-43d6-b75e-bf652ca28621","Type":"ContainerStarted","Data":"307189e74ba154e0e93c4c268e3b706f5c364d1851a77e1229911c1a92997903"} Nov 01 21:54:59.507677 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:54:59.507631 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podStartSLOduration=1.507618147 podStartE2EDuration="1.507618147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:59.50568457 +0000 UTC m=+646.479587100" watchObservedRunningTime="2026-11-01 21:54:59.507618147 +0000 UTC m=+646.481520720" Nov 01 21:55:00.067403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:00.067366 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:55:09.068412 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:09.068383 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:55:09.080960 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:09.080936 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:55:09.589801 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:09.589773 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:55:18.024060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.023992 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 21:55:18.029280 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.029259 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.037766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.037697 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ff3e70-openshift-default-dockercfg-8tnvh\"" Nov 01 21:55:18.042709 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.042685 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 21:55:18.071732 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.071700 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.071861 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.071745 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m8f\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.071861 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.071767 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.071861 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.071801 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.072003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.071983 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.072055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.072012 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.072097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.072085 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.072133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.072110 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.072166 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.072140 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173429 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173394 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173575 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173518 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173575 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173559 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m8f\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173673 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173588 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173673 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173626 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173720 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173740 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173764 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.173875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.173797 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.174104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.174077 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.174239 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.174206 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.174300 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.174240 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.174300 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.174251 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.174413 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.174347 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.176192 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.176170 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.176389 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.176372 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.185682 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.185655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.185763 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.185700 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m8f\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.343923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.343847 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:18.635371 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.635345 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 21:55:18.637019 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:55:18.636992 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247cb386_b126_4583_8b2f_cb0409a89d9b.slice/crio-8ec4def3004c50a13a73e1d156511a69898d3cdfc87ef37a3618c8c10e9dd049 WatchSource:0}: Error finding container 8ec4def3004c50a13a73e1d156511a69898d3cdfc87ef37a3618c8c10e9dd049: Status 404 returned error can't find the container with id 8ec4def3004c50a13a73e1d156511a69898d3cdfc87ef37a3618c8c10e9dd049 Nov 01 21:55:18.639724 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.639679 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:18.639802 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.639765 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:18.639841 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:18.639807 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:19.664472 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:19.664428 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" event={"ID":"247cb386-b126-4583-8b2f-cb0409a89d9b","Type":"ContainerStarted","Data":"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94"} Nov 01 21:55:19.665135 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:19.665095 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" event={"ID":"247cb386-b126-4583-8b2f-cb0409a89d9b","Type":"ContainerStarted","Data":"8ec4def3004c50a13a73e1d156511a69898d3cdfc87ef37a3618c8c10e9dd049"} Nov 01 21:55:19.702666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:19.702607 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podStartSLOduration=1.702593325 podStartE2EDuration="1.702593325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:19.699575 +0000 UTC m=+666.673477564" watchObservedRunningTime="2026-11-01 21:55:19.702593325 +0000 UTC m=+666.676495863" Nov 01 21:55:20.344348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:20.344313 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:20.345519 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:20.345500 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:20.345576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:20.345547 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 21:55:21.344433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:21.344395 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:21.344820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:21.344452 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 21:55:22.326819 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:55:22.326784 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:55:22.344667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:22.344637 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:22.345054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:22.344703 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 21:55:23.344479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:23.344448 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:23.344678 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:23.344508 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 21:55:24.344344 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:24.344267 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 21:55:24.344344 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:24.344326 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 21:55:25.350408 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:25.350375 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:25.350910 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:25.350811 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:25.351946 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:25.351923 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 21:55:46.417073 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.417043 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf"] Nov 01 21:55:46.421235 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.421216 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.427976 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.427951 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:55:46.429793 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.429772 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:55:46.431700 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.431647 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf"] Nov 01 21:55:46.555002 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.554966 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.555206 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.555020 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.555206 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.555135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbsn\" (UniqueName: \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.656395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.656362 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.656582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.656421 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.656582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.656449 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbsn\" (UniqueName: \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.657200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.657174 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.659176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.659151 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.669326 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.669270 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbsn\" (UniqueName: \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\") pod \"maas-api-revoke-sub-01fa4642-6wzbf\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:46.755682 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:46.755643 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:47.050819 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.050790 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf"] Nov 01 21:55:47.052352 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:55:47.052327 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31184b82_af95_480e_b003_033b5bd67260.slice/crio-c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f WatchSource:0}: Error finding container c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f: Status 404 returned error can't find the container with id c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f Nov 01 21:55:47.054414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.054398 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:55:47.596139 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.596111 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 21:55:47.601207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.601183 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.604957 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.604923 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f3a6b-openshift-default-dockercfg-nlg9b\"" Nov 01 21:55:47.612176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.612156 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 21:55:47.775082 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775052 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775240 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775087 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775240 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775131 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775318 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775249 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775375 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775358 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775410 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775390 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775413 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775432 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.775510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.775468 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.863891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.863815 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" event={"ID":"31184b82-af95-480e-b003-033b5bd67260","Type":"ContainerStarted","Data":"c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f"} Nov 01 21:55:47.876400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876369 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876688 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876411 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876688 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876453 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876800 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876714 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876791 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876836 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876875 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.876937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876912 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876931 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876939 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.876970 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.877318 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.877376 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.877675 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.877656 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.879070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.879047 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.879509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.879490 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.889097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.889068 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.889402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.889380 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:47.916223 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:47.916199 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:48.203875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.203847 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 21:55:48.206111 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:55:48.206081 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb99ef7_62ca_4357_bec2_09ef732562fb.slice/crio-2446f64271e864509e4f29d09195bf35e27cd5436bf709046c098221637c8525 WatchSource:0}: Error finding container 2446f64271e864509e4f29d09195bf35e27cd5436bf709046c098221637c8525: Status 404 returned error can't find the container with id 2446f64271e864509e4f29d09195bf35e27cd5436bf709046c098221637c8525 Nov 01 21:55:48.208741 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.208708 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:48.208843 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.208784 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:48.208843 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.208826 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:55:48.872154 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.872122 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" event={"ID":"2fb99ef7-62ca-4357-bec2-09ef732562fb","Type":"ContainerStarted","Data":"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac"} Nov 01 21:55:48.872609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.872177 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" event={"ID":"2fb99ef7-62ca-4357-bec2-09ef732562fb","Type":"ContainerStarted","Data":"2446f64271e864509e4f29d09195bf35e27cd5436bf709046c098221637c8525"} Nov 01 21:55:48.916365 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.916319 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:48.930416 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:48.930370 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podStartSLOduration=1.930357288 podStartE2EDuration="1.930357288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:48.927823475 +0000 UTC m=+695.901726013" watchObservedRunningTime="2026-11-01 21:55:48.930357288 +0000 UTC m=+695.904259881" Nov 01 21:55:49.919070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:49.919036 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:49.919668 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:49.919094 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:50.917131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:50.917051 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:50.917341 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:50.917318 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:50.923134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:50.923097 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" event={"ID":"31184b82-af95-480e-b003-033b5bd67260","Type":"ContainerStarted","Data":"9a42fdcbcb88d05bcc918feacc39bce3750eb595001caab6ab541be1df0a9b28"} Nov 01 21:55:50.941833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:50.941787 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" podStartSLOduration=1.8366948189999999 podStartE2EDuration="4.941776345s" totalImagesPullingTime="3.105081526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:55:50.940544947 +0000 UTC m=+697.914447524" watchObservedRunningTime="2026-11-01 21:55:50.941776345 +0000 UTC m=+697.915678884" Nov 01 21:55:51.937050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:51.936494 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:51.937050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:51.936566 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:51.946535 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:51.946504 4395 generic.go:356] "Generic (PLEG): container finished" podID="31184b82-af95-480e-b003-033b5bd67260" containerID="9a42fdcbcb88d05bcc918feacc39bce3750eb595001caab6ab541be1df0a9b28" exitCode=0 Nov 01 21:55:51.946680 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:51.946596 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" event={"ID":"31184b82-af95-480e-b003-033b5bd67260","Type":"ContainerDied","Data":"9a42fdcbcb88d05bcc918feacc39bce3750eb595001caab6ab541be1df0a9b28"} Nov 01 21:55:52.327318 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:55:52.327287 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:55:52.917010 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:52.916972 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:52.917226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:52.917054 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:53.136968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.136942 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:53.159427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.158209 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\") pod \"31184b82-af95-480e-b003-033b5bd67260\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " Nov 01 21:55:53.159427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.158393 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\") pod \"31184b82-af95-480e-b003-033b5bd67260\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " Nov 01 21:55:53.159427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.158490 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\" (UniqueName: \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\") pod \"31184b82-af95-480e-b003-033b5bd67260\" (UID: \"31184b82-af95-480e-b003-033b5bd67260\") " Nov 01 21:55:53.159427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.159108 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca" pod "31184b82-af95-480e-b003-033b5bd67260" (UID: "31184b82-af95-480e-b003-033b5bd67260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:55:53.164091 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.164063 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn" pod "31184b82-af95-480e-b003-033b5bd67260" (UID: "31184b82-af95-480e-b003-033b5bd67260"). InnerVolumeSpecName "kube-api-access-mpbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:55:53.166681 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.166660 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca" pod "31184b82-af95-480e-b003-033b5bd67260" (UID: "31184b82-af95-480e-b003-033b5bd67260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:55:53.259813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.259785 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31184b82-af95-480e-b003-033b5bd67260-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:53.259813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.259812 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbsn\" (UniqueName: \"kubernetes.io/projected/31184b82-af95-480e-b003-033b5bd67260-kube-api-access-mpbsn\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:53.260005 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.259823 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31184b82-af95-480e-b003-033b5bd67260-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:55:53.917439 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.917402 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:53.917694 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.917668 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:53.959545 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.959514 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" Nov 01 21:55:53.959545 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.959538 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf" event={"ID":"31184b82-af95-480e-b003-033b5bd67260","Type":"ContainerDied","Data":"c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f"} Nov 01 21:55:53.959751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:53.959631 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d5e8c649359d97af6f323f2a0b83460de85a18d328d5c5087ab7032bf22c1f" Nov 01 21:55:54.918056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:54.917642 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 21:55:54.918056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:54.917717 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 21:55:55.920962 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:55.920936 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:55.921487 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:55.921462 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:55:55.922220 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:55:55.922205 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 21:56:02.243262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.243228 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt"] Nov 01 21:56:02.243747 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.243735 4395 cpu_manager.go:414] "containerMap: removing container" podUID="31184b82-af95-480e-b003-033b5bd67260" containerName="revoke-keys" Nov 01 21:56:02.243808 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.243749 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31184b82-af95-480e-b003-033b5bd67260" containerName="revoke-keys" Nov 01 21:56:02.243842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.243814 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="31184b82-af95-480e-b003-033b5bd67260" containerName="revoke-keys" Nov 01 21:56:02.247133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.247115 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.253126 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.253101 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:56:02.253273 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.253209 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:56:02.256323 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.255907 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt"] Nov 01 21:56:02.262405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.262377 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcclt\" (UniqueName: \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.262525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.262452 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.262682 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.262659 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.363745 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.363717 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.363890 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.363805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.363890 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.363846 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcclt\" (UniqueName: \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.364329 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.364308 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.366554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.366535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.376268 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.376250 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcclt\" (UniqueName: \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\") pod \"maas-api-revoke-sub-76907f68-s2mtt\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.562381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.562346 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:02.848125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:02.848090 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt"] Nov 01 21:56:02.849698 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:02.849666 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19452e5d_eda5_40ce_a174_9d0f94db5c46.slice/crio-163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70 WatchSource:0}: Error finding container 163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70: Status 404 returned error can't find the container with id 163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70 Nov 01 21:56:03.030354 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:03.030316 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" event={"ID":"19452e5d-eda5-40ce-a174-9d0f94db5c46","Type":"ContainerStarted","Data":"163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70"} Nov 01 21:56:04.051019 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:04.050920 4395 generic.go:356] "Generic (PLEG): container finished" podID="19452e5d-eda5-40ce-a174-9d0f94db5c46" containerID="828223c8bd4fca16b5b6d531bd0d6be7eb28158fa104b2fe5cbc618ac04bf9b3" exitCode=0 Nov 01 21:56:04.051554 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:04.051121 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" event={"ID":"19452e5d-eda5-40ce-a174-9d0f94db5c46","Type":"ContainerDied","Data":"828223c8bd4fca16b5b6d531bd0d6be7eb28158fa104b2fe5cbc618ac04bf9b3"} Nov 01 21:56:05.231484 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.231460 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:05.291197 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.291155 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\") pod \"19452e5d-eda5-40ce-a174-9d0f94db5c46\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " Nov 01 21:56:05.291352 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.291237 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\" (UniqueName: \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\") pod \"19452e5d-eda5-40ce-a174-9d0f94db5c46\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " Nov 01 21:56:05.291352 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.291301 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\") pod \"19452e5d-eda5-40ce-a174-9d0f94db5c46\" (UID: \"19452e5d-eda5-40ce-a174-9d0f94db5c46\") " Nov 01 21:56:05.291571 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.291545 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca" pod "19452e5d-eda5-40ce-a174-9d0f94db5c46" (UID: "19452e5d-eda5-40ce-a174-9d0f94db5c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:56:05.291758 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.291734 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.293409 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.293382 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt" pod "19452e5d-eda5-40ce-a174-9d0f94db5c46" (UID: "19452e5d-eda5-40ce-a174-9d0f94db5c46"). InnerVolumeSpecName "kube-api-access-mcclt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:56:05.293518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.293497 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca" pod "19452e5d-eda5-40ce-a174-9d0f94db5c46" (UID: "19452e5d-eda5-40ce-a174-9d0f94db5c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:56:05.393285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.393206 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19452e5d-eda5-40ce-a174-9d0f94db5c46-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:05.393285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:05.393240 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcclt\" (UniqueName: \"kubernetes.io/projected/19452e5d-eda5-40ce-a174-9d0f94db5c46-kube-api-access-mcclt\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:56:06.063115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:06.063088 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" Nov 01 21:56:06.063115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:06.063114 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt" event={"ID":"19452e5d-eda5-40ce-a174-9d0f94db5c46","Type":"ContainerDied","Data":"163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70"} Nov 01 21:56:06.063474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:06.063155 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163443a9061d23d64533b548dd5b36eea36fc0b29f64520848b9120ab2cbbf70" Nov 01 21:56:17.944923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.944893 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 21:56:17.946015 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.945902 4395 cpu_manager.go:414] "containerMap: removing container" podUID="19452e5d-eda5-40ce-a174-9d0f94db5c46" containerName="revoke-keys" Nov 01 21:56:17.946015 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.945925 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19452e5d-eda5-40ce-a174-9d0f94db5c46" containerName="revoke-keys" Nov 01 21:56:17.946258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.946070 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="19452e5d-eda5-40ce-a174-9d0f94db5c46" containerName="revoke-keys" Nov 01 21:56:17.951287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.951266 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:17.958557 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.958214 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b0b7a3-openshift-default-dockercfg-rxv7b\"" Nov 01 21:56:17.963188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:17.963166 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.115972 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116060 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116136 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116188 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116215 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116237 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116260 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116345 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svms\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.117564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.116422 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.217761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217688 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.217761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217745 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.217953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217858 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.217953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217924 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217969 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.217991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218014 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218076 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218166 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svms\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218195 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218233 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218356 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218454 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.218914 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.218890 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.220586 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.220556 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.220908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.220888 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.230305 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.230286 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.230389 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.230315 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svms\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.267244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.267218 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:18.552842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.552819 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 21:56:18.554231 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:18.554199 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3963ac0_ee4e_49ba_9dc0_4c76692b8176.slice/crio-d1cbf786a609fbfee11c22a028a0b965cabded6da1233e20c8aac36bba53e108 WatchSource:0}: Error finding container d1cbf786a609fbfee11c22a028a0b965cabded6da1233e20c8aac36bba53e108: Status 404 returned error can't find the container with id d1cbf786a609fbfee11c22a028a0b965cabded6da1233e20c8aac36bba53e108 Nov 01 21:56:18.557122 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.557092 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:18.557193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.557155 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:18.557193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:18.557186 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:19.253694 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.253661 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 21:56:19.257991 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.257960 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.264796 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.264772 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f3a6b\"/\"distinct-w0-1f3a6b-kserve-self-signed-certs\"" Nov 01 21:56:19.269541 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.269506 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 21:56:19.297799 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.297774 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" event={"ID":"c3963ac0-ee4e-49ba-9dc0-4c76692b8176","Type":"ContainerStarted","Data":"05cc74160f957677e50da61e64cebd842103217efe59e401dec975b0508dd44a"} Nov 01 21:56:19.297970 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.297825 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" event={"ID":"c3963ac0-ee4e-49ba-9dc0-4c76692b8176","Type":"ContainerStarted","Data":"d1cbf786a609fbfee11c22a028a0b965cabded6da1233e20c8aac36bba53e108"} Nov 01 21:56:19.321778 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.321736 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podStartSLOduration=2.321723607 podStartE2EDuration="2.321723607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:19.3193952 +0000 UTC m=+726.293297738" watchObservedRunningTime="2026-11-01 21:56:19.321723607 +0000 UTC m=+726.295626219" Nov 01 21:56:19.331489 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.331464 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.331604 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.331557 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.331866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.331846 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.331979 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.331963 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.332080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.332061 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rw7\" (UniqueName: \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.332222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.332206 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447238 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447341 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447429 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447468 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447507 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rw7\" (UniqueName: \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.447585 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.448083 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.448591 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.448304 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.452405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.449755 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.490433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.489826 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rw7\" (UniqueName: \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.491001 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.490948 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.491001 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.490985 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\") pod \"distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.580101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.578267 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:19.901902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:19.901864 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 21:56:19.902913 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:19.902881 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aaee67e_3a32_4532_ae18_6adf1da6a58b.slice/crio-4e1e421ffe17740832906a574e8f9d5f6e0656a8eebfe1e4c3e7af80d0f2af08 WatchSource:0}: Error finding container 4e1e421ffe17740832906a574e8f9d5f6e0656a8eebfe1e4c3e7af80d0f2af08: Status 404 returned error can't find the container with id 4e1e421ffe17740832906a574e8f9d5f6e0656a8eebfe1e4c3e7af80d0f2af08 Nov 01 21:56:20.273056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.267957 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:20.273056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.269411 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:20.273056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.269474 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:20.312804 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.312771 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" event={"ID":"2aaee67e-3a32-4532-ae18-6adf1da6a58b","Type":"ContainerStarted","Data":"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8"} Nov 01 21:56:20.313083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.313066 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" event={"ID":"2aaee67e-3a32-4532-ae18-6adf1da6a58b","Type":"ContainerStarted","Data":"4e1e421ffe17740832906a574e8f9d5f6e0656a8eebfe1e4c3e7af80d0f2af08"} Nov 01 21:56:20.338475 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.338421 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podStartSLOduration=1.338406757 podStartE2EDuration="1.338406757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:20.33543929 +0000 UTC m=+727.309341831" watchObservedRunningTime="2026-11-01 21:56:20.338406757 +0000 UTC m=+727.312309296" Nov 01 21:56:20.592107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:20.583107 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:21.276291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:21.276257 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:21.276915 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:21.276873 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:22.279522 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:22.279070 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:22.279522 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:22.279128 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:22.445532 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:22.445492 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:56:22.472095 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:56:22.460180 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:56:22.480056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:22.477114 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:23.360586 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:23.357586 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:23.360586 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:23.357642 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:24.350401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:24.350363 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:24.350660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:24.350631 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:24.454160 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:24.454119 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:24.582656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:24.582138 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:56:25.272230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:25.272194 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:25.272507 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:25.272479 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:26.267968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:26.267939 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 21:56:26.268455 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:26.267991 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 21:56:27.926191 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:27.926160 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:27.926665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:27.926571 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:27.927689 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:27.927664 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 21:56:29.579681 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:29.579647 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:29.592671 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:29.592649 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:29.884108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:29.882519 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:56:30.403800 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.403764 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 21:56:30.407839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.407822 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.413831 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.413805 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ff3e70\"/\"distinct-w3-ff3e70-kserve-self-signed-certs\"" Nov 01 21:56:30.419197 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.419172 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 21:56:30.529013 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.528976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8c8\" (UniqueName: \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.529221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.529162 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.529221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.529194 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.529315 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.529236 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.529370 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.529345 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.529490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.529465 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632622 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632714 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8c8\" (UniqueName: \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632854 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632876 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632949 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.632995 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.636183 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.636742 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.639050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.636974 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.640419 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.640319 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.645418 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.645396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.648893 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.648869 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8c8\" (UniqueName: \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\") pod \"distinct-w3-ff3e70-kserve-564f777f9c-cxk97\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:30.726504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:30.726435 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:31.014142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:31.014113 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 21:56:31.014822 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:31.014792 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc61b9164_97c2_4eeb_ab5b_f44c4a7e08b9.slice/crio-45abd67df1df39c7320b491f10f99b9bbaf18326a756aeb9d5d14bd3dd467870 WatchSource:0}: Error finding container 45abd67df1df39c7320b491f10f99b9bbaf18326a756aeb9d5d14bd3dd467870: Status 404 returned error can't find the container with id 45abd67df1df39c7320b491f10f99b9bbaf18326a756aeb9d5d14bd3dd467870 Nov 01 21:56:31.883612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:31.883580 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" event={"ID":"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9","Type":"ContainerStarted","Data":"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f"} Nov 01 21:56:31.883993 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:31.883618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" event={"ID":"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9","Type":"ContainerStarted","Data":"45abd67df1df39c7320b491f10f99b9bbaf18326a756aeb9d5d14bd3dd467870"} Nov 01 21:56:31.904160 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:31.904114 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podStartSLOduration=1.904101432 podStartE2EDuration="1.904101432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:31.901680632 +0000 UTC m=+738.875583173" watchObservedRunningTime="2026-11-01 21:56:31.904101432 +0000 UTC m=+738.878003971" Nov 01 21:56:32.727079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.727050 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:32.772770 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.772739 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 21:56:32.776815 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.776794 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.780644 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.780624 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb9597-openshift-default-dockercfg-lhqww\"" Nov 01 21:56:32.790313 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.790285 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 21:56:32.874003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.873973 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874195 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874054 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874195 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874143 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874244 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874327 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874290 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874327 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874308 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874448 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874432 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874514 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874472 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.874565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.874508 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrw8\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.975985 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.975948 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.975995 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976012 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976121 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976155 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976187 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrw8\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976232 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976310 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976345 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976920 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976529 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976920 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976537 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976920 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976819 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.976920 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.976817 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.977198 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.977132 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.978873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.978839 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.979257 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.979234 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.989679 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.989656 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:32.989783 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:32.989745 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrw8\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:33.092498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.092476 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:33.428604 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.428574 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 21:56:33.430148 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:33.430104 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0891c_109e_46b5_ae68_fec2aa6d4b44.slice/crio-1dd4384912cbede290a1eee2885ec3f18e5fd0501aae4465206bed1c7503500d WatchSource:0}: Error finding container 1dd4384912cbede290a1eee2885ec3f18e5fd0501aae4465206bed1c7503500d: Status 404 returned error can't find the container with id 1dd4384912cbede290a1eee2885ec3f18e5fd0501aae4465206bed1c7503500d Nov 01 21:56:33.432897 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.432867 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:33.432996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.432937 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:33.432996 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.432969 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:33.895651 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.895617 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" event={"ID":"23d0891c-109e-46b5-ae68-fec2aa6d4b44","Type":"ContainerStarted","Data":"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f"} Nov 01 21:56:33.895811 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.895679 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" event={"ID":"23d0891c-109e-46b5-ae68-fec2aa6d4b44","Type":"ContainerStarted","Data":"1dd4384912cbede290a1eee2885ec3f18e5fd0501aae4465206bed1c7503500d"} Nov 01 21:56:33.921625 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:33.921582 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podStartSLOduration=1.921570099 podStartE2EDuration="1.921570099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:33.919099416 +0000 UTC m=+740.893001956" watchObservedRunningTime="2026-11-01 21:56:33.921570099 +0000 UTC m=+740.895472716" Nov 01 21:56:34.092758 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:34.092734 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:34.094097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:34.094073 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:34.094159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:34.094129 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:35.099490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:35.099454 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:35.099985 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:35.099527 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:36.097055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:36.093434 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:36.097055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:36.093496 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:37.096387 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:37.096351 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:37.097117 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:37.097087 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:38.144283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:38.144249 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:38.144923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:38.144890 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:39.097418 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:39.097385 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:39.097710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:39.097684 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:40.097497 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:40.097458 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 01 21:56:40.113154 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:40.113111 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 01 21:56:40.726618 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:40.726581 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:40.740323 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:40.740294 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:40.993971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:40.993883 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:56:41.097178 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:41.097149 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:41.097443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:41.097424 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:41.098153 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:41.098132 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 21:56:47.802813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.802776 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 21:56:47.807947 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.807927 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.819341 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.819317 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 21:56:47.885205 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885177 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdm5q\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885363 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885227 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885363 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885287 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885363 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885343 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885540 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885470 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885540 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885495 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885577 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885601 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.885691 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.885633 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.986923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.986883 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.987251 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.987229 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.987371 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.987347 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.987863 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.987913 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.987940 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.988012 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdm5q\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.988376 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.988416 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.988449 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.989579 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.989555 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.990256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.989839 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.990256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.989939 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.995377 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.995344 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:47.999178 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:47.999132 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:48.001000 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.000953 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:48.008324 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.008293 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:48.008595 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.008574 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdm5q\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:48.123768 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.123703 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:48.412856 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.412706 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 21:56:48.415939 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:48.415908 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376beac4_30fa_443f_8345_5d9a0d85ac94.slice/crio-bf5109f9cdf144308d5e2274e29e98e63b66c022dadd592902bf25e72c9b9cd7 WatchSource:0}: Error finding container bf5109f9cdf144308d5e2274e29e98e63b66c022dadd592902bf25e72c9b9cd7: Status 404 returned error can't find the container with id bf5109f9cdf144308d5e2274e29e98e63b66c022dadd592902bf25e72c9b9cd7 Nov 01 21:56:48.418609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.418567 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:48.418690 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.418654 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:48.418738 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:48.418696 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:56:49.046587 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.046551 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" event={"ID":"376beac4-30fa-443f-8345-5d9a0d85ac94","Type":"ContainerStarted","Data":"b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278"} Nov 01 21:56:49.047078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.046610 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" event={"ID":"376beac4-30fa-443f-8345-5d9a0d85ac94","Type":"ContainerStarted","Data":"bf5109f9cdf144308d5e2274e29e98e63b66c022dadd592902bf25e72c9b9cd7"} Nov 01 21:56:49.072668 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.072625 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podStartSLOduration=2.072610281 podStartE2EDuration="2.072610281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:49.069304654 +0000 UTC m=+756.043207205" watchObservedRunningTime="2026-11-01 21:56:49.072610281 +0000 UTC m=+756.046512820" Nov 01 21:56:49.124400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.124371 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:49.125824 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.125800 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:49.125933 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:49.125845 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:50.125071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.125012 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:50.125445 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.125085 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:50.607415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.607383 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 21:56:50.620345 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.614739 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 21:56:50.620345 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.614852 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.625262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.625237 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"distinct-2-w6-fb9597-kserve-self-signed-certs\"" Nov 01 21:56:50.627442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.627423 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"kube-root-ca.crt\"" Nov 01 21:56:50.627656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.627645 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"default-dockercfg-77f4g\"" Nov 01 21:56:50.627813 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.627483 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"openshift-service-ca.crt\"" Nov 01 21:56:50.642649 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642627 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.642753 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642678 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.642830 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642758 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.642830 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmx5s\" (UniqueName: \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.642927 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642880 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.642966 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.642931 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.756058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.749178 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.756058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.749473 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.756058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.749690 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmx5s\" (UniqueName: \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.756058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.749918 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.756058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.750185 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.765078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.760162 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.765078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.764287 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.765078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.764458 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.765078 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.764686 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.765928 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.765904 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.801411 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.801384 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.802454 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.802430 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmx5s\" (UniqueName: \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\") pod \"distinct-2-w6-fb9597-kserve-6f94988978-zmgc6\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:50.950708 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:50.946340 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:51.124515 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:51.124483 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:51.124677 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:51.124539 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:51.304502 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:51.304457 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 21:56:51.304834 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:56:51.304564 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15349852_76c5_49b2_9a02_385f2d225f17.slice/crio-deaa2ef44f060a1ed52d313d7fa11f74423cd317b6bec609ebeb771bdf4047ce WatchSource:0}: Error finding container deaa2ef44f060a1ed52d313d7fa11f74423cd317b6bec609ebeb771bdf4047ce: Status 404 returned error can't find the container with id deaa2ef44f060a1ed52d313d7fa11f74423cd317b6bec609ebeb771bdf4047ce Nov 01 21:56:52.067107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.067066 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" event={"ID":"15349852-76c5-49b2-9a02-385f2d225f17","Type":"ContainerStarted","Data":"2576d0f5066dcdc772ac265f2cec109cf7038901aa4944592a4211a2befe3c04"} Nov 01 21:56:52.067107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.067110 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" event={"ID":"15349852-76c5-49b2-9a02-385f2d225f17","Type":"ContainerStarted","Data":"deaa2ef44f060a1ed52d313d7fa11f74423cd317b6bec609ebeb771bdf4047ce"} Nov 01 21:56:52.099151 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.095009 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podStartSLOduration=2.094992256 podStartE2EDuration="2.094992256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:52.091382474 +0000 UTC m=+759.065285014" watchObservedRunningTime="2026-11-01 21:56:52.094992256 +0000 UTC m=+759.068894797" Nov 01 21:56:52.124247 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.124219 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:52.124403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.124278 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:52.325201 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:56:52.325123 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:56:52.952565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:52.952535 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:56:53.124742 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:53.124712 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:53.124922 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:53.124759 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:54.124479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:54.124444 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 21:56:54.124900 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:54.124491 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 21:56:55.458362 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:55.458321 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:55.458867 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:55.458802 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:56:55.459698 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:56:55.459680 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 21:57:00.949860 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:00.949825 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:00.962097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:00.962071 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:01.167089 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:01.167065 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:02.707704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.707676 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 21:57:02.712304 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.712281 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.728301 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.728275 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 21:57:02.843906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.843871 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.843906 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.843909 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844166 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844007 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844166 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844123 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844166 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844148 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844257 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x68t\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844289 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.844516 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.844403 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945333 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945301 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945416 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945474 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945664 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945502 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x68t\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945664 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945537 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945664 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945587 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945876 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945729 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945876 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945757 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.945876 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945760 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.946069 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.945730 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.946179 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.946142 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.946246 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.946220 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.946304 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.946283 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.948167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.948142 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.948264 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.948234 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.962504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.962204 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:02.963849 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:02.963507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x68t\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:03.033621 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.033590 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 21:57:03.039307 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.039278 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:03.063807 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.063777 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 21:57:03.064012 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.063910 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.124510 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.124478 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 21:57:03.129608 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.129582 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.148584 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.148559 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 21:57:03.156948 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.156806 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.156948 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.156901 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.156954 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.156997 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zsw\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.157094 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.157125 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.157183 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.157253 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.157338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.157283 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.210913 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.210883 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 21:57:03.215641 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.215495 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.237055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.230650 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 21:57:03.258542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258518 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.258666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258556 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.258666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258595 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.258666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258643 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.258883 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258861 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.258954 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258894 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.258954 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258926 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.258954 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258951 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259135 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258981 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259135 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.258996 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259251 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259202 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259251 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259207 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259367 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259349 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxdr\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259418 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259377 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259418 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259389 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259418 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259406 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259595 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259517 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259652 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259554 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.259780 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259670 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259780 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259722 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.259780 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259765 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zsw\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.260046 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259795 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.260046 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.259673 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.261494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.261468 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.262294 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.262267 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.271564 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.271539 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.271641 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.271622 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zsw\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.355375 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.355352 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 21:57:03.357273 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:03.357251 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea09ab9_9560_4d24_9af0_f7520a375323.slice/crio-32847cd1f9b38050787e05f494219382bae0c7179d6cfe3e1e573489f7fd54e7 WatchSource:0}: Error finding container 32847cd1f9b38050787e05f494219382bae0c7179d6cfe3e1e573489f7fd54e7: Status 404 returned error can't find the container with id 32847cd1f9b38050787e05f494219382bae0c7179d6cfe3e1e573489f7fd54e7 Nov 01 21:57:03.359746 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.359717 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.359821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.359806 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.359870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.359858 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.360733 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360707 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxdr\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.360840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360774 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.360840 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360801 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.360964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360840 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.360964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360873 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.360974 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361173 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361242 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361297 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361274 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361303 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361436 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361537 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361537 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361506 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361543 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361619 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361653 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361675 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361720 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361756 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd57f\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.361912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361828 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.361912 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.361855 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.362156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.362002 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.362156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.362133 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.364129 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.364111 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.365283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.365266 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.373673 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.373652 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxdr\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.374312 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.374286 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.401853 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.401834 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:03.463169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463132 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463345 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463195 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463345 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463274 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd57f\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463345 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463413 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463439 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463605 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463486 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463605 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463509 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463605 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.463908 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.463884 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.464096 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.464321 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.464761 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.465777 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.467784 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.473378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.472533 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.485277 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.474278 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.485428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.478430 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd57f\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.502127 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.502104 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.536194 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.536162 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:03.709889 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.709865 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 21:57:03.712708 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:03.712676 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c72e686_3ea0_40f7_98b5_5b1ba24eb5c0.slice/crio-d0005797060fced441bd19c333976aae46b2f8c96c87725b8bc8e1dc716368c6 WatchSource:0}: Error finding container d0005797060fced441bd19c333976aae46b2f8c96c87725b8bc8e1dc716368c6: Status 404 returned error can't find the container with id d0005797060fced441bd19c333976aae46b2f8c96c87725b8bc8e1dc716368c6 Nov 01 21:57:03.718041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.717987 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.718243 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.718218 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.718414 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.718394 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.815264 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:03.815223 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6529c812_e3d7_46b9_9320_05958ff0580b.slice/crio-e0569730de380c16ff2bb019eb5d22503b50ad2e9ec35715f08925317c88fec6 WatchSource:0}: Error finding container e0569730de380c16ff2bb019eb5d22503b50ad2e9ec35715f08925317c88fec6: Status 404 returned error can't find the container with id e0569730de380c16ff2bb019eb5d22503b50ad2e9ec35715f08925317c88fec6 Nov 01 21:57:03.815561 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.815533 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 21:57:03.819123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.819088 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.819200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.819165 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.819246 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.819208 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.857639 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.857612 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 21:57:03.864269 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.864237 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.864464 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.864439 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:03.864572 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:03.864489 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:04.188882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.188847 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" event={"ID":"40362c5d-e7d8-408e-8961-71efe6d7f16f","Type":"ContainerStarted","Data":"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91"} Nov 01 21:57:04.189236 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.189210 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" event={"ID":"40362c5d-e7d8-408e-8961-71efe6d7f16f","Type":"ContainerStarted","Data":"e03fd62a95e49e1940be0079a6b2e7b7aab6948268f59fb6caf46bc485bb29d7"} Nov 01 21:57:04.206963 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.206929 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" event={"ID":"6529c812-e3d7-46b9-9320-05958ff0580b","Type":"ContainerStarted","Data":"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671"} Nov 01 21:57:04.207242 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.207221 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" event={"ID":"6529c812-e3d7-46b9-9320-05958ff0580b","Type":"ContainerStarted","Data":"e0569730de380c16ff2bb019eb5d22503b50ad2e9ec35715f08925317c88fec6"} Nov 01 21:57:04.218699 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.218665 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" event={"ID":"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0","Type":"ContainerStarted","Data":"b960aeb53472fb592840b1859c83604a39306e207d051a85bd5388763e290380"} Nov 01 21:57:04.233604 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.218928 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" event={"ID":"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0","Type":"ContainerStarted","Data":"d0005797060fced441bd19c333976aae46b2f8c96c87725b8bc8e1dc716368c6"} Nov 01 21:57:04.237615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.236995 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podStartSLOduration=1.236976018 podStartE2EDuration="1.236976018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.234453472 +0000 UTC m=+771.208356012" watchObservedRunningTime="2026-11-01 21:57:04.236976018 +0000 UTC m=+771.210878560" Nov 01 21:57:04.239708 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.239681 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" event={"ID":"8ea09ab9-9560-4d24-9af0-f7520a375323","Type":"ContainerStarted","Data":"a6d531ef6500114cdd33149a1d155d694176cbb8be27485865088958ee5544f5"} Nov 01 21:57:04.239834 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.239732 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" event={"ID":"8ea09ab9-9560-4d24-9af0-f7520a375323","Type":"ContainerStarted","Data":"32847cd1f9b38050787e05f494219382bae0c7179d6cfe3e1e573489f7fd54e7"} Nov 01 21:57:04.261205 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.261092 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podStartSLOduration=2.261074892 podStartE2EDuration="2.261074892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.257890234 +0000 UTC m=+771.231792785" watchObservedRunningTime="2026-11-01 21:57:04.261074892 +0000 UTC m=+771.234977433" Nov 01 21:57:04.316055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.315567 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podStartSLOduration=1.31554339 podStartE2EDuration="1.31554339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.298435937 +0000 UTC m=+771.272338501" watchObservedRunningTime="2026-11-01 21:57:04.31554339 +0000 UTC m=+771.289445930" Nov 01 21:57:04.337055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.329777 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podStartSLOduration=2.329757529 podStartE2EDuration="2.329757529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:04.323109815 +0000 UTC m=+771.297012346" watchObservedRunningTime="2026-11-01 21:57:04.329757529 +0000 UTC m=+771.303660068" Nov 01 21:57:04.402506 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.402423 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:04.405282 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.404928 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:04.405282 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.404984 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:04.468011 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.467966 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:04.469695 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.469662 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:04.469823 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.469716 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:04.551580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:04.551419 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:05.119053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.101381 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:05.119053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.115163 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.119053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.115247 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:05.405606 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.405511 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.405890 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.405857 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:05.469051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.468249 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.469051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.468320 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:05.541350 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.541303 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:05.541636 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:05.541594 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:06.087067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.046426 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.087067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.046484 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:06.412108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.411445 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.412108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.411540 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:06.443338 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.443298 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.443700 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.443572 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.469498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.469444 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.470859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.470832 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.471060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.471016 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:06.542569 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.542535 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:06.542887 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.542840 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:06.582222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.581111 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.866052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.864160 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:06.979182 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:06.979135 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.148727 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.148640 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.152127 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.152091 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:07.421228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.403411 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.421801 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.421738 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:07.512226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.499297 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.534054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.528759 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.582112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.557226 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.582112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.557288 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:07.608126 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.582459 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.608126 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.590157 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:07.608126 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.590208 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:07.762229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.756655 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:57:07.762229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.756717 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:57:07.762229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.756763 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:57:07.885531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.885478 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:07.887080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:07.887056 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:57:08.040869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.040789 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.041255 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.041228 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:08.193734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.193693 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:08.444237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.444126 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.444237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.444184 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.552146 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.552232 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.552723 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.552768 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.553679 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:08.559051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.554048 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:57:08.595053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.588993 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:08.959184 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:08.959121 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.048446 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.048408 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.050230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.050194 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:09.436053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.435575 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.436053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.435634 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:09.469008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.468965 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.469617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.469568 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:09.537230 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.537197 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:09.537526 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.537489 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:09.559622 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.559587 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.571050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.559813 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.675971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.675515 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.982049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.981174 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:09.984055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:09.981650 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.044277 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.044243 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.044577 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.044533 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:10.072118 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.072068 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.172055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.159489 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.408080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.404394 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.408080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.404457 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:10.466799 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.466757 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.467134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.467100 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:10.568191 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.568140 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.580121 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.579967 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:10.580121 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.580044 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:10.776206 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.776156 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.776829 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.776788 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:10.783627 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.783601 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:57:10.991308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.985115 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:10.991308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:10.985812 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:57:11.121270 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.121182 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.121270 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.121247 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:11.186702 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.186648 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:11.379120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.378788 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:11.450056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.449964 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.450056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.450044 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 21:57:11.455048 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.451667 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:57:11.474142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.469256 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.474142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.469321 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 21:57:11.474142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.472910 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:57:11.480474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.480451 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:11.534055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.531075 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:57:11.564828 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.564788 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:11.565012 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.564849 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:11.568254 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:11.568225 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:57:12.041117 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.041081 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.041328 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.041147 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 21:57:12.537082 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.537044 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 21:57:12.537355 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.537326 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 21:57:12.568825 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.568788 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:12.570777 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.570749 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:12.581862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:12.581829 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 21:57:13.101702 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.101380 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:13.109406 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.109381 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:13.110663 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.110641 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 21:57:13.342765 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.342737 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:13.343802 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.343649 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:13.346249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.346227 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 21:57:13.541329 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.541300 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:13.541631 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.541610 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:13.542191 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:13.542172 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 21:57:17.939101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.939064 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 21:57:17.945398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.945372 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.954392 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.954318 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"distinct-2-w1-53b784-kserve-self-signed-certs\"" Nov 01 21:57:17.968017 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.967985 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 21:57:17.989405 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989379 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qsx\" (UniqueName: \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.989584 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989413 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.989642 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989597 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.989691 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989649 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.989733 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989719 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:17.989781 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:17.989746 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.142310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.142268 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qsx\" (UniqueName: \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.198449 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.198658 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.198743 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.198818 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.198860 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.199560 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.199801 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.201674 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.203225 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 21:57:18.208857 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.208723 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.222757 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.222729 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 21:57:18.235181 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.235148 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qsx\" (UniqueName: \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.252061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.235936 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.284436 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.256445 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\") pod \"distinct-2-w1-53b784-kserve-88f8d7576-2vnml\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.402325 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402334 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4c6v\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402498 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402447 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402696 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402825 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402806 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402885 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402874 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402896 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.402934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402915 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.403020 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.402962 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.503825 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.503798 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.503967 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.503927 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504074 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.503985 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504146 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504087 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504146 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504120 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504147 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504174 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504205 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504443 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504421 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504427 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504576 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504484 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4c6v\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504684 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504649 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504684 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504651 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.504795 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.504693 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.506707 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.506685 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.506802 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.506720 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.516465 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.516443 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.516616 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.516598 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4c6v\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.577830 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.577807 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:18.597811 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.597785 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:18.863726 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:18.863697 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode59111f7_0000_4ee7_9647_1c9571db616b.slice/crio-69af22712ada0ff262d3b573c09bee2a313f24d7da9eb1c8651f93aea15f2d87 WatchSource:0}: Error finding container 69af22712ada0ff262d3b573c09bee2a313f24d7da9eb1c8651f93aea15f2d87: Status 404 returned error can't find the container with id 69af22712ada0ff262d3b573c09bee2a313f24d7da9eb1c8651f93aea15f2d87 Nov 01 21:57:18.863867 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.863779 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 21:57:18.890123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.890096 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 21:57:18.892320 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:18.892293 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77599b7b_6134_4adf_8462_bde0e3d20b24.slice/crio-38b90a8ee4ba3efa2bd11b405e3f6dea88b12c83b63943517004eaab8fbbbcd0 WatchSource:0}: Error finding container 38b90a8ee4ba3efa2bd11b405e3f6dea88b12c83b63943517004eaab8fbbbcd0: Status 404 returned error can't find the container with id 38b90a8ee4ba3efa2bd11b405e3f6dea88b12c83b63943517004eaab8fbbbcd0 Nov 01 21:57:18.894960 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.894923 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:18.895046 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.895008 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:18.895095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:18.895074 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:19.855816 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.855777 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" event={"ID":"77599b7b-6134-4adf-8462-bde0e3d20b24","Type":"ContainerStarted","Data":"b6df81d7b852d637b8f2fc2ebd5439df41fc95cd8beb3b3b30f1f15998741023"} Nov 01 21:57:19.856340 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.855842 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" event={"ID":"77599b7b-6134-4adf-8462-bde0e3d20b24","Type":"ContainerStarted","Data":"38b90a8ee4ba3efa2bd11b405e3f6dea88b12c83b63943517004eaab8fbbbcd0"} Nov 01 21:57:19.857584 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.857558 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" event={"ID":"e59111f7-0000-4ee7-9647-1c9571db616b","Type":"ContainerStarted","Data":"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab"} Nov 01 21:57:19.857584 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.857592 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" event={"ID":"e59111f7-0000-4ee7-9647-1c9571db616b","Type":"ContainerStarted","Data":"69af22712ada0ff262d3b573c09bee2a313f24d7da9eb1c8651f93aea15f2d87"} Nov 01 21:57:19.903097 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.902995 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podStartSLOduration=1.90297684 podStartE2EDuration="1.90297684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:19.89239447 +0000 UTC m=+786.866297022" watchObservedRunningTime="2026-11-01 21:57:19.90297684 +0000 UTC m=+786.876879379" Nov 01 21:57:19.914834 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:19.914784 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podStartSLOduration=2.914764273 podStartE2EDuration="2.914764273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:19.912995719 +0000 UTC m=+786.886898263" watchObservedRunningTime="2026-11-01 21:57:19.914764273 +0000 UTC m=+786.888666811" Nov 01 21:57:20.578248 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:20.578220 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:20.600573 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:20.600536 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:20.608055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:20.606365 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:20.608055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:20.606416 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:21.599406 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:21.599371 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:21.599874 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:21.599427 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:22.440162 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:57:22.440123 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:57:22.598859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:22.598412 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:22.598859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:22.598474 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:23.601052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:23.600760 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:23.601052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:23.600846 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:24.486308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.485142 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:24.532052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.525541 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:57:24.570160 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.570116 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:24.571523 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.571491 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:24.590185 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.590147 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:24.614651 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.614624 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:24.615210 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:24.615185 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:25.012772 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.011714 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.555138 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.534094 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.571878 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.571849 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:25.572276 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.572246 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.598052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.574623 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:57:25.611060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.601839 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:25.611060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.602244 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:25.611060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.602569 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:57:25.611060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.602711 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:57:25.647082 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.640362 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:57:25.649788 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.649768 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:57:25.671482 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.671450 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:25.671609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.671510 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:25.672454 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.672433 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:57:25.673283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.673263 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:57:25.690523 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.690505 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:25.756049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.754259 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:57:25.807547 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:25.807176 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:57:26.598495 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:26.598455 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 01 21:57:26.598659 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:26.598522 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 01 21:57:27.602880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:27.602851 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:27.603402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:27.603380 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:27.604319 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:27.604289 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 21:57:28.583042 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:28.582988 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:28.634107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:28.634079 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:29.131151 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.131126 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:29.739512 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.739480 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 21:57:29.744728 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.744699 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.753237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.753206 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 21:57:29.753384 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.753324 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"kube-root-ca.crt\"" Nov 01 21:57:29.753384 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.753333 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"default-dockercfg-pc87z\"" Nov 01 21:57:29.753499 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.753402 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"openshift-service-ca.crt\"" Nov 01 21:57:29.754081 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.754062 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"distinct-2-w2-f033b1-kserve-self-signed-certs\"" Nov 01 21:57:29.795817 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.795786 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.795989 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.795847 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.796110 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.796083 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.796178 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.796135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.796234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.796212 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sm5x\" (UniqueName: \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.796287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.796267 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.904944 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.904904 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.905717 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.905692 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.916383 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.916311 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.938563 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.938533 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.939212 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.939188 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sm5x\" (UniqueName: \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.947093 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.939803 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.947916 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.906182 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.948122 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.917045 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.948244 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.938440 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.948351 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.947635 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:29.975612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:29.975578 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:30.003427 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:30.000624 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sm5x\" (UniqueName: \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\") pod \"distinct-2-w2-f033b1-kserve-f88cd889-z4xqp\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:30.066388 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:30.066352 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:30.390761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:30.387731 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 21:57:31.139539 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:31.139507 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" event={"ID":"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1","Type":"ContainerStarted","Data":"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e"} Nov 01 21:57:31.139896 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:31.139546 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" event={"ID":"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1","Type":"ContainerStarted","Data":"743634a450b144f9d1b612931f1ccd33c8ed1de15bfceebdd7d5a5f7e79b7d7b"} Nov 01 21:57:31.160173 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:31.160116 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podStartSLOduration=2.16010049 podStartE2EDuration="2.16010049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:31.15842411 +0000 UTC m=+798.132326646" watchObservedRunningTime="2026-11-01 21:57:31.16010049 +0000 UTC m=+798.134003029" Nov 01 21:57:32.067287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:32.067253 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:40.066869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:40.066837 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:40.080665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:40.080636 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:40.275729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:40.275703 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:47.916798 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.916762 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 21:57:47.922956 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.922936 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.946075 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.946044 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 21:57:47.981360 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981329 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981372 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981504 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981636 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981736 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981715 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981815 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981857 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.981891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.981886 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:47.982042 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:47.982002 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kh7\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.073239 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.073200 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 21:57:48.077408 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.077384 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.083048 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083001 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083188 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083078 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083200 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083333 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083255 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083333 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083295 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083336 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kh7\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083367 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083404 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083425 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083665 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083535 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083757 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083734 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083873 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.083895 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083888 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.084011 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.083963 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.086075 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.086048 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.086302 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.086284 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.092735 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.092709 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 21:57:48.102156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.102129 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kh7\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.105266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.105244 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.115673 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.115647 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 21:57:48.122690 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.122672 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.131873 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.131847 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 21:57:48.185565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185490 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfnt\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.185565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185536 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.185755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185625 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.185755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185670 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.185755 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185747 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.185898 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185771 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.185953 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.185896 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.186048 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186017 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.186116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186069 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186116 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186110 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xpv\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186189 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186228 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.186306 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186295 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.186341 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186323 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186373 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186345 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.186417 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186403 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.186453 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.186430 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.248507 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.248474 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:48.267320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.267296 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 21:57:48.272862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.272844 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295692 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295756 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295786 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295849 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfnt\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295881 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295935 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.295995 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296098 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296135 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296225 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296340 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296432 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296464 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xpv\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296519 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296564 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296662 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.296722 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.299041 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.297309 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.304042 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.302494 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.304426 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.304397 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.305603 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.305550 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.305992 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.305968 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.306882 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.306859 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.309079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.309015 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 21:57:48.311380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.311350 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.311648 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.311624 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.312500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.312471 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.312798 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.312773 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.313532 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.313500 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.315801 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.315765 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.318730 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.318246 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 21:57:48.318730 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.318408 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.318730 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.318700 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.330666 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.330145 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.335698 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.334709 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.337083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.337009 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfnt\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.338491 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.338460 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 21:57:48.341845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.341766 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.342058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.342015 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xpv\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.398354 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398319 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398530 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398462 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.398530 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398511 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398544 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398577 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwh9\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.398656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398607 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.398656 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398653 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398698 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvtw\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398727 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.398875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398806 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.398875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398843 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.398875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398873 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.399152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398910 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.399152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.398959 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.399152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.399065 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.399152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.399105 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.399152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.399137 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.399459 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.399235 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.442062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.441977 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:48.453093 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.453066 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501199 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501253 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501339 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501402 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501492 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501522 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501554 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501593 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwh9\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501627 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501669 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501716 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvtw\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501749 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501832 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501878 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501913 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.501956 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.502013 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.502105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.502109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.506566 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.506282 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.507207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.506811 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.507207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.507107 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.507521 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.507655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.507710 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.507975 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.508068 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.508286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.508245 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.508626 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.508462 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.509724 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.509152 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.509724 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.509463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.509724 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.509687 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.510387 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.510196 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.528536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.525721 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwh9\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.534186 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.533906 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:48.534186 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.533904 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.535535 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.535483 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvtw\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.565716 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.565660 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 21:57:48.568936 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.568904 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:48.569086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.568969 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:48.569086 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.568996 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:48.634320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.634290 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:48.670075 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:48.661337 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:49.013079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.002498 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.013079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.002599 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.013079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.002645 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.068823 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.030073 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 21:57:49.179131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.138631 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.179131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.138784 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.179131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.138865 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:49.294050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.291692 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 21:57:49.406369 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.403560 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" event={"ID":"f28479be-0e23-471b-bb40-61976623ba89","Type":"ContainerStarted","Data":"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb"} Nov 01 21:57:49.406369 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.403620 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" event={"ID":"f28479be-0e23-471b-bb40-61976623ba89","Type":"ContainerStarted","Data":"7b11832f145fbcd4a3a83fae97939a7a4f2e7f9a468945a2f554bba25f82f910"} Nov 01 21:57:49.443736 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.443671 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" event={"ID":"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293","Type":"ContainerStarted","Data":"5e8505473afe6af7c987f31a855aa7ca6eb75aa775c7945f904d499a020643cc"} Nov 01 21:57:49.464257 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.464156 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" event={"ID":"359f99ba-ea97-4f56-9553-9e1ea84c10ee","Type":"ContainerStarted","Data":"afce06068c7f9a11d32495e893be664ff459cb17cedcd0cdf6c7421d087ab3c4"} Nov 01 21:57:49.495150 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:49.495087 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podStartSLOduration=2.495064067 podStartE2EDuration="2.495064067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:49.443338136 +0000 UTC m=+816.417240676" watchObservedRunningTime="2026-11-01 21:57:49.495064067 +0000 UTC m=+816.468966627" Nov 01 21:57:50.264592 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.258584 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 21:57:50.288077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.271843 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:57:50.288077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.272766 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:50.288077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.272812 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:50.377505 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:57:50.374455 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e474bb8_1d3e_4bab_8514_1931f8bcc3cf.slice/crio-0c2ffefc676dbacf5a2dfe54b44b65d5651a7c0d1d273f208041398ef69064ee WatchSource:0}: Error finding container 0c2ffefc676dbacf5a2dfe54b44b65d5651a7c0d1d273f208041398ef69064ee: Status 404 returned error can't find the container with id 0c2ffefc676dbacf5a2dfe54b44b65d5651a7c0d1d273f208041398ef69064ee Nov 01 21:57:50.410951 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.384051 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:50.410951 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.384140 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:50.410951 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.384225 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:50.696632 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.696594 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" event={"ID":"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf","Type":"ContainerStarted","Data":"0c2ffefc676dbacf5a2dfe54b44b65d5651a7c0d1d273f208041398ef69064ee"} Nov 01 21:57:50.721315 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.721273 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:50.818265 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:50.818231 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 21:57:51.125061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.112362 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 01 21:57:51.266542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.141791 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:57:51.344697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.284467 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.432556 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.284767 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:51.432851 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.432799 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:51.557064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.555329 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:51.557064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.555496 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:51.557064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.555588 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:57:51.622072 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.618047 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.699045 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.698985 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.939907 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.742142 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:51.939907 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.742504 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.965145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.786424 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:51.965145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.786469 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:57:51.965145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.851494 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:51.965145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:51.940527 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:57:52.048551 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.048509 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 01 21:57:52.186054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.166338 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:57:52.516055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.354651 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:52.516055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.354725 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:52.516055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.355788 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:52.516055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.367977 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:57:52.577058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.556783 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:57:52.586018 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.585982 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:52.909053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.908625 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" event={"ID":"9f995556-a12a-4021-950a-7130a1467e7e","Type":"ContainerStarted","Data":"1cb1b422df1314e557ce6a507104f6196b8d295dd3ff68326dab00485ad6a20e"} Nov 01 21:57:52.938176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.938128 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.939895 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.938464 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.970411 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.938431 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.943576 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:52.943611 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.006900 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:57:53.104538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.073135 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:57:53.254566 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.213088 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:57:53.304386 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.232081 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" podUID="a389fbce-17d4-4613-884d-37aa85377992" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.304861 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.304092 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:53.305692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.304999 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:53.389552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.365521 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:53.982117 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.811535 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.043054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.962254 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.043054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.965609 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.043054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:53.996767 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:54.043054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.033888 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:54.115007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.114500 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.347237 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.347190 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:57:54.542291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.390546 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:54.542291 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:57:54.486085 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:57:54.542291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.504202 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:57:54.542291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.504249 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" event={"ID":"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293","Type":"ContainerStarted","Data":"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e"} Nov 01 21:57:54.542291 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.523545 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:57:54.623142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.598629 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.623142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.598684 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:54.718936 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.718899 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:54.723259 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.723226 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:54.735435 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:54.729918 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" event={"ID":"359f99ba-ea97-4f56-9553-9e1ea84c10ee","Type":"ContainerStarted","Data":"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4"} Nov 01 21:57:55.013891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.013866 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:55.050660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.050603 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:57:55.061108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.056388 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:55.061108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.057645 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:55.061803 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.061755 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.122235 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.122178 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.411552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.411444 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.422702 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.422666 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:55.495205 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.495172 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.495446 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.495419 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:57:55.509113 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.497116 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:55.616722 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.616682 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:55.616965 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.505839 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:57:55.647785 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.505809 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:55.921052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.911438 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podStartSLOduration=7.911414218 podStartE2EDuration="7.911414218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:54.989696313 +0000 UTC m=+821.963598855" watchObservedRunningTime="2026-11-01 21:57:55.911414218 +0000 UTC m=+822.885316758" Nov 01 21:57:55.932272 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:55.932238 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:57:56.050234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.050175 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.064224 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.064170 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.094094 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.068103 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:56.126065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.068158 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:56.139885 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.126383 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.248105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.247407 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podStartSLOduration=8.247379296 podStartE2EDuration="8.247379296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:55.12337689 +0000 UTC m=+822.097279430" watchObservedRunningTime="2026-11-01 21:57:56.247379296 +0000 UTC m=+823.221281836" Nov 01 21:57:56.535106 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.524408 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 01 21:57:56.778299 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.778247 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.806139 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.797227 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:56.815052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.807604 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.868056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.860375 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:56.929053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.860323 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" event={"ID":"9f995556-a12a-4021-950a-7130a1467e7e","Type":"ContainerStarted","Data":"38c317f4265cf8edbb3d86a82f1ef79281436ce5bb17fce1f4389fd642f4b2ff"} Nov 01 21:57:56.954208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.816240 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:57:56.954511 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.954475 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:57:56.954677 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:56.931061 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.062725 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.062506 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.089266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.089222 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:57:57.116055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.066985 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.116055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.092555 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:57:57.116055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.092839 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:57.116634 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.116589 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:57:57.181057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.138139 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:57:57.181057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.161303 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" event={"ID":"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf","Type":"ContainerStarted","Data":"3ce1029ea6d277b32e5a8805b141d2a2e85e20eb43accfae3ae3ecf7addf38c5"} Nov 01 21:57:57.270055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.231663 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:57.295053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.281216 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.295053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.281306 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:57.475582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.475475 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.481196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.481155 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:57.521056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.496145 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podStartSLOduration=9.496122455 podStartE2EDuration="9.496122455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:57.002453713 +0000 UTC m=+823.976356253" watchObservedRunningTime="2026-11-01 21:57:57.496122455 +0000 UTC m=+824.470024995" Nov 01 21:57:57.565101 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.502516 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.565363 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.565330 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:57:57.652571 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.639352 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:57:57.679061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.662202 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.683628 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.683165 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:57:57.741119 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.716237 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:57:57.775428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.720587 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podStartSLOduration=9.720563105 podStartE2EDuration="9.720563105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:57.687110264 +0000 UTC m=+824.661012801" watchObservedRunningTime="2026-11-01 21:57:57.720563105 +0000 UTC m=+824.694465645" Nov 01 21:57:57.797380 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.797333 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:57.823177 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.723146 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:57:57.831154 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.831113 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:57:57.831429 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.798555 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.004108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:57.965615 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.097729 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.097682 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:57:58.124053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.123897 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.167087 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.145581 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.239050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.230318 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.251374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.251339 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.251635 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.251603 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:58.295866 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.295772 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:57:58.446425 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.446389 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.446676 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.446646 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:58.459046 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.459002 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.459301 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.459276 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:57:58.641613 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.641519 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.642147 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.642103 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:57:58.688172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.688128 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:57:58.688481 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.688429 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:57:58.828171 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.828122 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:58.838343 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:58.838301 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.069607 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.067889 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.104186 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.104121 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.129043 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.128976 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.170323 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.169132 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.231168 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.231116 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.249335 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.249300 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.249649 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.249606 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:57:59.306054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.303134 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.453500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.453404 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.453764 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.453731 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:57:59.454987 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.454963 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.455178 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.455158 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:57:59.580505 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.580149 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.643381 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.643347 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.643686 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.643646 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:57:59.689580 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.689538 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:57:59.689894 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.689849 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:57:59.833052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.829123 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:57:59.867211 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:57:59.867112 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:00.068933 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.068879 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.069570 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.069532 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 21:58:00.114476 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.114371 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.146287 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.145148 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:00.169574 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.169528 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.245243 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.245182 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.257339 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.257291 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.257613 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.257572 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:58:00.303503 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.303461 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.464125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.463231 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.464125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.463295 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:58:00.464125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.463919 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.464125 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.463957 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:58:00.586179 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.586125 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.650139 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.640649 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.650402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.650370 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:58:00.688891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.688849 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:58:00.689182 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.689151 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:58:00.840204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.840155 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:00.878245 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:00.878189 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:01.081655 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.081612 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:01.107113 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.082251 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.120176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.120126 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.153345 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.153302 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.180574 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.177708 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:01.246377 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.246316 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.262933 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.262269 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.262933 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.262336 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:58:01.308180 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.308128 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.464120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.463164 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.464120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.463232 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:58:01.464120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.464094 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.464440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.464140 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:58:01.535055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.534105 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 21:58:01.636512 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.636477 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.636785 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.636741 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:58:01.691067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.689424 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:58:01.691067 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.689485 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:58:01.745211 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.745107 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 21:58:01.849181 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.849131 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:01.889167 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.889119 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:01.965216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:01.965170 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 01 21:58:02.083175 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.083125 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.130176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.130125 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.160179 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.160124 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.177981 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.177937 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.247435 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.247390 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.307518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.250868 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.307828 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.307795 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:58:02.312131 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.312098 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.449262 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.449169 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.449500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.449472 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:58:02.459054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.456711 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.459054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.456757 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:58:02.643617 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.643580 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.644350 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.644316 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:58:02.694442 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.694404 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:58:02.694752 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.694713 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:58:02.863049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.861994 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:02.890371 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:02.890322 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.125120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.092124 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.139643 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.139216 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.249055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.221552 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.249055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.221886 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:03.249321 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.249229 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:58:03.250609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.250437 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.250609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.250485 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 21:58:03.339988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.338451 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:03.339988 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.338865 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:03.443207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.443120 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:58:03.449334 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.449304 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.449552 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.449529 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:58:03.459448 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.459271 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.459448 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.459315 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 21:58:03.464054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.458116 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:58:03.523052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.522357 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:58:03.540537 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.540509 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:58:03.697234 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.697133 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:58:03.717565 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.717518 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:58:03.732087 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.719432 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.732087 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.719482 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:58:03.742820 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.742795 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:58:03.746151 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.746117 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:58:03.862343 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.862301 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:58:03.984403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:03.984317 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:58:04.155206 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.155158 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.232004 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.231961 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.232599 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.232568 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:04.249108 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.241423 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:58:04.375212 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.375168 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:04.505871 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.503963 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.505871 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.504018 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 21:58:04.548538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.548505 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:58:04.638848 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.638756 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.639139 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.639112 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 21:58:04.715053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.705223 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 21:58:04.715053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.705293 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 21:58:04.770358 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.770281 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:58:04.886533 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.883961 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:58:04.943121 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.941473 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:58:04.946415 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:04.946322 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 21:58:05.010474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.010444 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:58:05.107317 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.107216 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:58:05.117073 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.116319 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 21:58:05.455207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.455112 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:58:05.457998 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.457967 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 21:58:05.838447 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.838415 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:58:05.841320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:05.841290 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 21:58:06.087311 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:06.087282 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:58:06.089083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:06.089005 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 21:58:17.862596 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.862558 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 21:58:17.869207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.869183 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.897295 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.897239 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 21:58:17.936583 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936552 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936743 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936626 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8wv\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936743 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936718 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936748 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936842 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936838 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936856 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.936937 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.936892 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.937084 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.937051 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:17.937235 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:17.937214 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.036530 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.036309 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 21:58:18.047275 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047229 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047605 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8wv\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047663 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047696 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047793 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047823 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047867 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.047922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.048076 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.049421 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.049009 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.051155 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.051126 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.052183 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.052150 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.052835 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.052819 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.054644 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.054623 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.059193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.054898 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.073543 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.073470 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.074346 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.074276 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.077134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.077113 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.078474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.077806 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8wv\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.088171 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.080887 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 21:58:18.149494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149458 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.149710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149690 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbd4p\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.149710 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149722 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.149868 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149815 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.149964 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149943 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.150010 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.149978 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.150092 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.150011 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.150228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.150208 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.150347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.150333 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.225226 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.225204 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:18.252826 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.252800 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253150 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253130 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253266 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253251 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253356 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253340 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbd4p\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253420 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253400 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253500 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253485 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253588 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253572 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.253647 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.253634 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.254112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.254071 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.254646 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.254609 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.255194 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.255135 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.255394 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.255342 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.255770 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.255723 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.257008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.256980 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.259667 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.259629 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.261278 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.261253 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.270553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.270534 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.270875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.270851 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbd4p\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.411831 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.411744 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:18.528845 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.528811 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 21:58:18.530142 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:58:18.530098 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02eff61_15a6_42e0_98ed_aeef6ffb8c45.slice/crio-184de348c0c72ca439818e273fb747c5632eabf0d616db95510cdc63be458197 WatchSource:0}: Error finding container 184de348c0c72ca439818e273fb747c5632eabf0d616db95510cdc63be458197: Status 404 returned error can't find the container with id 184de348c0c72ca439818e273fb747c5632eabf0d616db95510cdc63be458197 Nov 01 21:58:18.533091 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.533059 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.533208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.533138 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.533208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.533168 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.714102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.714070 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 21:58:18.720645 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.720600 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.721185 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.720956 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.721296 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.721230 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:18.935567 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.934574 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" event={"ID":"f02eff61-15a6-42e0-98ed-aeef6ffb8c45","Type":"ContainerStarted","Data":"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a"} Nov 01 21:58:18.935567 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:18.934637 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" event={"ID":"f02eff61-15a6-42e0-98ed-aeef6ffb8c45","Type":"ContainerStarted","Data":"184de348c0c72ca439818e273fb747c5632eabf0d616db95510cdc63be458197"} Nov 01 21:58:19.012102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.010953 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" event={"ID":"66a04d67-3da0-4336-b86a-b13d2701b947","Type":"ContainerStarted","Data":"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73"} Nov 01 21:58:19.012102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.011016 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" event={"ID":"66a04d67-3da0-4336-b86a-b13d2701b947","Type":"ContainerStarted","Data":"d9402646f129bbbd5847fe304552a1b04a474764cef2b3ff0c67865c417fe277"} Nov 01 21:58:19.026398 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.026329 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podStartSLOduration=2.026307949 podStartE2EDuration="2.026307949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:18.96628171 +0000 UTC m=+845.940184269" watchObservedRunningTime="2026-11-01 21:58:19.026307949 +0000 UTC m=+846.000210490" Nov 01 21:58:19.225596 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.225562 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:19.227479 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.227453 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:19.227612 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.227506 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:19.412520 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.412415 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:19.414164 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.414142 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:19.414249 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:19.414212 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:20.230052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:20.229828 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:20.230052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:20.229889 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:20.414531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:20.414492 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:20.414815 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:20.414787 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:21.231392 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:21.231360 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:21.232003 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:21.231975 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:21.416449 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:21.416412 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:21.416739 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:21.416708 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:22.234881 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:22.234749 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:22.235540 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:22.235510 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:22.364870 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:58:22.364841 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:58:22.429536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:22.429186 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:22.429536 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:22.429265 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:23.238444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:23.238355 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:23.239146 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:23.238441 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:23.434432 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:23.434395 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:23.434713 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:23.434682 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:24.304055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:24.265476 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:24.304055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:24.265543 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:24.497285 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:24.497201 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:24.497542 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:24.497516 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:25.154054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.149767 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:25.171435 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.169874 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.171435 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.170730 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.329088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.288565 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:25.329088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.288633 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:25.329088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.307147 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:25.339256 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.338681 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.413478 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.413386 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:25.413744 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.413698 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:25.453059 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.437175 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:25.652331 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:25.651131 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:58:26.128821 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.128779 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.158054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.154130 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.181553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.181520 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:58:26.201088 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.183009 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.252056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.196878 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.346085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.272093 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:26.346085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.272158 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:26.346085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.273755 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podStartSLOduration=8.273735473 podStartE2EDuration="8.273735473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.054552574 +0000 UTC m=+846.028455114" watchObservedRunningTime="2026-11-01 21:58:26.273735473 +0000 UTC m=+853.247638014" Nov 01 21:58:26.346085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.343264 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.346918 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.311021 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.348170 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.348141 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:58:26.378399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.368610 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:58:26.378399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.369210 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:58:26.378399 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.369267 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:58:26.458112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.432875 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.458112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.433414 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:26.458112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.433454 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:26.522488 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.522429 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:26.524073 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:26.524051 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:58:27.340055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.204653 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:27.487325 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.399884 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:27.488087 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.314322 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:58:27.540081 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.380406 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:27.565805 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.418691 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.566166 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.189359 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:58:27.566375 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.535162 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.566524 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.566497 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:27.566718 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.420786 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:27.729106 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.728422 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:27.729106 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.728496 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:27.813056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:27.782957 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.045123 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.041730 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:58:28.176053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.141695 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:58:28.368112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.319384 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:28.368112 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.319443 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:28.409271 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.409222 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.513054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.488063 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:28.537111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.496881 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:28.537111 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.524299 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:28.537473 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.537434 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.582053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.577287 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:28.582053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.578601 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:28.582053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.580312 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:58:28.582053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.581616 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.658200 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.658112 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:28.658456 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.658419 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.849172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.845140 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:28.849172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.845214 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.861191 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.861146 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:28.913016 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:28.898559 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:58:29.077164 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.077124 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:29.077428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.077402 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.371168 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.371118 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.371428 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.371398 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:29.434647 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.434599 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 21:58:29.435149 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.435102 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 21:58:29.441902 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.441878 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:58:29.573197 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.573142 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.628354 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.628247 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:58:29.629659 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.629620 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 01 21:58:29.677176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.671580 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:29.697771 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.697721 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:29.900154 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.876223 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:29.920141 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.920100 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:58:29.973462 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.973431 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:58:29.979582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:29.979555 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:58:30.022339 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.022309 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:58:30.160692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.148472 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:58:30.183297 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.183268 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:58:30.248053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.245410 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 21:58:30.248053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.245473 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 21:58:30.258493 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.258325 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:58:30.963060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.962592 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:30.970071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.964407 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:30.989050 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:30.985205 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 21:58:32.269152 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.269121 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:32.269690 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.269169 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:32.349374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.349344 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:32.349719 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.349696 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:32.350485 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.350468 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 21:58:32.609689 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.609589 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv"] Nov 01 21:58:32.614012 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.613989 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.618096 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.618021 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 21:58:32.618208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.618150 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 21:58:32.622833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.622808 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv"] Nov 01 21:58:32.685460 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.685428 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.685626 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.685565 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9dm\" (UniqueName: \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.685690 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.685675 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.795839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.787170 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.795839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.787448 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.795839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.787605 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9dm\" (UniqueName: \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.795839 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.794171 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.825401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.824863 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9dm\" (UniqueName: \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.825401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.825352 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ec4b36-2gwvv\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.898551 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.898465 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 21:58:32.903357 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.903336 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.914611 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.914587 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 21:58:32.934677 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.934642 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:32.991983 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.991945 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.992172 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.992012 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.992242 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.992210 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993133 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993107 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993273 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993258 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993350 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993335 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993441 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993475 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993454 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlr77\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:32.993512 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:32.993471 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095083 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095120 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095126 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlr77\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095324 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095291 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095452 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095433 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095490 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095527 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095653 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095549 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095653 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095577 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095887 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095664 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095887 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095778 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.095887 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095800 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.096105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095884 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.096105 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.095911 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.096501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.096480 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.098568 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.098515 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.098800 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.098780 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.107136 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.107116 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlr77\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.107900 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.107880 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.221748 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.221663 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:33.235992 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.235963 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv"] Nov 01 21:58:33.238588 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:58:33.238555 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24ea02b_5ba7_4868_b2d3_e7fa36c21842.slice/crio-f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6 WatchSource:0}: Error finding container f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6: Status 404 returned error can't find the container with id f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6 Nov 01 21:58:33.405495 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.405454 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" event={"ID":"f24ea02b-5ba7-4868-b2d3-e7fa36c21842","Type":"ContainerStarted","Data":"f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6"} Nov 01 21:58:33.525481 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.525452 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 21:58:33.527236 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:58:33.527205 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8a5c4_19bf_41b0_a20c_b773c93a02dc.slice/crio-42ab6a9f1d77cf85de44a171c90c0c56bcf227c9371c8568be4fbfa591d2133b WatchSource:0}: Error finding container 42ab6a9f1d77cf85de44a171c90c0c56bcf227c9371c8568be4fbfa591d2133b: Status 404 returned error can't find the container with id 42ab6a9f1d77cf85de44a171c90c0c56bcf227c9371c8568be4fbfa591d2133b Nov 01 21:58:33.530081 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.530009 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:33.530187 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.530118 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:33.530187 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:33.530148 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:58:34.477626 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:34.477216 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" event={"ID":"eda8a5c4-19bf-41b0-a20c-b773c93a02dc","Type":"ContainerStarted","Data":"e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd"} Nov 01 21:58:34.477626 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:34.477291 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" event={"ID":"eda8a5c4-19bf-41b0-a20c-b773c93a02dc","Type":"ContainerStarted","Data":"42ab6a9f1d77cf85de44a171c90c0c56bcf227c9371c8568be4fbfa591d2133b"} Nov 01 21:58:34.481219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:34.480093 4395 generic.go:356] "Generic (PLEG): container finished" podID="f24ea02b-5ba7-4868-b2d3-e7fa36c21842" containerID="251e65fe730c0d5aaad54334a3bb5faaa304695ccd8f7a513af0616eb719ca48" exitCode=0 Nov 01 21:58:34.481219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:34.480302 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" event={"ID":"f24ea02b-5ba7-4868-b2d3-e7fa36c21842","Type":"ContainerDied","Data":"251e65fe730c0d5aaad54334a3bb5faaa304695ccd8f7a513af0616eb719ca48"} Nov 01 21:58:34.518069 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:34.517993 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podStartSLOduration=2.517977102 podStartE2EDuration="2.517977102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:34.508529395 +0000 UTC m=+861.482431934" watchObservedRunningTime="2026-11-01 21:58:34.517977102 +0000 UTC m=+861.491879641" Nov 01 21:58:35.306052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.305435 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:35.306052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.305491 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:35.308074 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.307243 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:35.750058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.750015 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:35.820320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.820274 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\") pod \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " Nov 01 21:58:35.820494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.820368 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\" (UniqueName: \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\") pod \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " Nov 01 21:58:35.820588 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.820533 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\") pod \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\" (UID: \"f24ea02b-5ba7-4868-b2d3-e7fa36c21842\") " Nov 01 21:58:35.821275 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.820857 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca" pod "f24ea02b-5ba7-4868-b2d3-e7fa36c21842" (UID: "f24ea02b-5ba7-4868-b2d3-e7fa36c21842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:58:35.821275 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.821246 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:35.823385 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.823360 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm" pod "f24ea02b-5ba7-4868-b2d3-e7fa36c21842" (UID: "f24ea02b-5ba7-4868-b2d3-e7fa36c21842"). InnerVolumeSpecName "kube-api-access-4t9dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:58:35.823629 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.823598 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca" pod "f24ea02b-5ba7-4868-b2d3-e7fa36c21842" (UID: "f24ea02b-5ba7-4868-b2d3-e7fa36c21842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:58:35.922811 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.922725 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:35.922811 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:35.922754 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9dm\" (UniqueName: \"kubernetes.io/projected/f24ea02b-5ba7-4868-b2d3-e7fa36c21842-kube-api-access-4t9dm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:58:36.228440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:36.228305 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:36.228440 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:36.228358 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:36.518179 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:36.518145 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" event={"ID":"f24ea02b-5ba7-4868-b2d3-e7fa36c21842","Type":"ContainerDied","Data":"f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6"} Nov 01 21:58:36.518352 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:36.518196 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f645e4eba983d06740c9c875f07b284b20e2be2a93f7cba17adce44141c576c6" Nov 01 21:58:36.518352 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:36.518201 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv" Nov 01 21:58:37.231270 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:37.225975 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:37.231270 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:37.226052 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:38.232007 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:38.231967 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:38.232538 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:38.232063 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:39.234060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:39.223204 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:39.234060 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:39.223266 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:40.223057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:40.222726 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 21:58:40.223057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:40.222795 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 21:58:41.395990 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:41.395960 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:41.396465 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:41.396445 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:41.397555 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:41.397536 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 21:58:52.395946 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:58:52.395905 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:58:55.550053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:55.546128 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:55.601062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:55.597119 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:55.845914 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:55.844109 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:55.928064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:55.921109 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:55.994862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:55.994121 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.409147 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.409103 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.436199 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.436163 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 01 21:58:56.518193 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.518154 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.547182 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.547140 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.548640 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.548615 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:58:56.590016 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.589974 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.622061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.599458 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:56.622061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.600109 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:58:56.898184 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.898136 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.940057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.932715 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:58:56.940057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.933873 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:56.940057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:56.934274 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:58:57.021055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.020385 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:58:57.072063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.021650 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:58:57.460223 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.460180 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:57.572124 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.569237 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.625476 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.602124 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 21:58:57.650864 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.650205 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:58:57.680288 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.656624 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:58:57.755204 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.755179 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:58:57.755490 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.755474 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:58:57.755660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.755647 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:58:57.759055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.758177 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:57.759243 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.759217 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:57.925406 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.925361 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:58:57.954474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:57.954139 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.032228 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.032126 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.474273 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.474174 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:58:58.577858 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.577822 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:58:58.630077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.615121 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.663203 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.663158 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.722356 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.711747 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:58.795368 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.792744 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:58:58.973054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:58.969370 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:58:59.012688 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.012644 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.141130 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.139586 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 21:58:59.417199 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.417101 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:58:59.417471 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.417446 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.480207 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.480161 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.583258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.583195 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:58:59.584370 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.584326 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.673547 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.673461 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:58:59.678991 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.678967 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:58:59.682437 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.682354 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:58:59.687521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.687105 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:58:59.705075 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.705019 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:58:59.709216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.709176 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:58:59.709216 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.709219 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:58:59.718290 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:58:59.718262 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:59:02.742493 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.742442 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:02.743501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.743418 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f24ea02b-5ba7-4868-b2d3-e7fa36c21842" containerName="revoke-keys" Nov 01 21:59:02.743501 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.743436 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24ea02b-5ba7-4868-b2d3-e7fa36c21842" containerName="revoke-keys" Nov 01 21:59:02.744002 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.743642 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24ea02b-5ba7-4868-b2d3-e7fa36c21842" containerName="revoke-keys" Nov 01 21:59:02.749276 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.749254 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.753637 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.753611 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-93a347eb-openshift-default-dockercfg-t5kdb\"" Nov 01 21:59:02.764347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.764323 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957594 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957647 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957698 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957781 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957829 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957930 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gmh\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.958057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.957987 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.984057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.976292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:02.984057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:02.983374 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.084898 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.084855 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gmh\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.085221 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.085197 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.085385 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.085369 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.085641 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.085626 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.085847 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.085832 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.085950 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.085936 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.086099 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.086081 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.086278 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.086263 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.086457 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.086430 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.087635 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.087594 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.089322 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.089293 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.089777 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.089731 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.090149 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.090105 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.090347 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.090324 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.104919 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.104817 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.130047 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.116631 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.141052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.132220 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.141052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.136407 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gmh\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\") pod \"e2e-ait-93a347eb-openshift-default-95778769c-7g26j\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.367892 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.367796 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:03.664715 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.664683 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:03.665249 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:03.665222 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7337c1f2_8f6f_4a7a_9f8f_b63b6cfb8008.slice/crio-f1081ddb0f2e0e3bc7c80399aa43f1dd894df45ae77e915a106b1168fc36b3a3 WatchSource:0}: Error finding container f1081ddb0f2e0e3bc7c80399aa43f1dd894df45ae77e915a106b1168fc36b3a3: Status 404 returned error can't find the container with id f1081ddb0f2e0e3bc7c80399aa43f1dd894df45ae77e915a106b1168fc36b3a3 Nov 01 21:59:03.668342 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.668309 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:03.668449 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.668372 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:03.668449 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.668403 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:03.769541 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.769510 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" event={"ID":"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008","Type":"ContainerStarted","Data":"87127a558c91895b582e1365f8693438ee8f3a68e399661e7f1e1d01a4c36493"} Nov 01 21:59:03.769870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.769572 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" event={"ID":"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008","Type":"ContainerStarted","Data":"f1081ddb0f2e0e3bc7c80399aa43f1dd894df45ae77e915a106b1168fc36b3a3"} Nov 01 21:59:03.797633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:03.794439 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" podStartSLOduration=1.794422456 podStartE2EDuration="1.794422456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:03.79143079 +0000 UTC m=+890.765333326" watchObservedRunningTime="2026-11-01 21:59:03.794422456 +0000 UTC m=+890.768324996" Nov 01 21:59:04.381049 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:04.375343 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:04.404056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:04.401397 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:04.776749 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:04.776716 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:04.777523 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:04.777506 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:05.419582 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.419548 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:05.423932 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.423915 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.428940 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.428898 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-13364262-dockercfg-2mkm8\"" Nov 01 21:59:05.437310 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.436884 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:05.449346 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.449325 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-13364262\"" Nov 01 21:59:05.706107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.705336 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddsx\" (UniqueName: \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.706107 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.705440 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.850053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.841009 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddsx\" (UniqueName: \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.850053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.841167 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.850053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.842433 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:05.962134 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:05.943773 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddsx\" (UniqueName: \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\") pod \"payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:06.053850 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.053815 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:06.338297 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.338266 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:06.338509 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:06.338484 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b1497f_cf6d_4baa_a838_06e7b276415e.slice/crio-9cf146b0ae63787f1c08a0bef1dc16438a73e96042d4355805e9ca70c3fe0949 WatchSource:0}: Error finding container 9cf146b0ae63787f1c08a0bef1dc16438a73e96042d4355805e9ca70c3fe0949: Status 404 returned error can't find the container with id 9cf146b0ae63787f1c08a0bef1dc16438a73e96042d4355805e9ca70c3fe0949 Nov 01 21:59:06.573141 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.573112 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:06.577627 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.577606 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.591258 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.591233 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:06.654052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.647372 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-681db477-dockercfg-gblmm\"" Nov 01 21:59:06.654052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.647462 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-681db477\"" Nov 01 21:59:06.679102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.673603 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prszg\" (UniqueName: \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.679102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.673841 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.802054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.801440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.802054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.801751 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prszg\" (UniqueName: \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.814875 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.814825 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.828597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.828564 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prszg\" (UniqueName: \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\") pod \"payload-processing-e2e-ait-681db477-64946f668d-mf6dm\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:06.900635 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:06.900322 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:07.097079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:07.096356 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" event={"ID":"33b1497f-cf6d-4baa-a838-06e7b276415e","Type":"ContainerStarted","Data":"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163"} Nov 01 21:59:07.097079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:07.096479 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:07.097079 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:07.096512 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" event={"ID":"33b1497f-cf6d-4baa-a838-06e7b276415e","Type":"ContainerStarted","Data":"9cf146b0ae63787f1c08a0bef1dc16438a73e96042d4355805e9ca70c3fe0949"} Nov 01 21:59:07.119810 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:07.119180 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" podStartSLOduration=2.119158962 podStartE2EDuration="2.119158962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:07.115812953 +0000 UTC m=+894.089715498" watchObservedRunningTime="2026-11-01 21:59:07.119158962 +0000 UTC m=+894.093061502" Nov 01 21:59:07.443424 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:07.443392 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:07.443706 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:07.443684 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b5aeb6_c463_4e3a_8779_d924c51e00c0.slice/crio-076a6921c9c78c3cd6d8b80c7340f3a9be168c907c865eb74facff73f7dda75a WatchSource:0}: Error finding container 076a6921c9c78c3cd6d8b80c7340f3a9be168c907c865eb74facff73f7dda75a: Status 404 returned error can't find the container with id 076a6921c9c78c3cd6d8b80c7340f3a9be168c907c865eb74facff73f7dda75a Nov 01 21:59:08.124547 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:08.124500 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" event={"ID":"88b5aeb6-c463-4e3a-8779-d924c51e00c0","Type":"ContainerStarted","Data":"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5"} Nov 01 21:59:08.125104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:08.124567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" event={"ID":"88b5aeb6-c463-4e3a-8779-d924c51e00c0","Type":"ContainerStarted","Data":"076a6921c9c78c3cd6d8b80c7340f3a9be168c907c865eb74facff73f7dda75a"} Nov 01 21:59:08.125104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:08.124627 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:08.150373 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:08.150319 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" podStartSLOduration=2.150302211 podStartE2EDuration="2.150302211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:08.14523428 +0000 UTC m=+895.119136832" watchObservedRunningTime="2026-11-01 21:59:08.150302211 +0000 UTC m=+895.124204754" Nov 01 21:59:15.567051 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.566496 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:15.613259 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.612804 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.634378 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.632214 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:15.653070 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.644586 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-93a347eb-serving-cert\"" Nov 01 21:59:15.763587 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.762295 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54vb\" (UniqueName: \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.763587 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.762615 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.763587 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.762702 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.763587 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.762781 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.878159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.872338 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.878159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.872437 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.878159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.872508 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.878159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.872588 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54vb\" (UniqueName: \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.878159 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.874419 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.887633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.887597 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.888181 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.888150 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.892940 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.892911 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54vb\" (UniqueName: \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\") pod \"maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:15.977279 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:15.977247 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:16.411838 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:16.411391 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:17.385419 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.385382 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" event={"ID":"be971324-fc2d-4b3c-b593-ebeb28f1de25","Type":"ContainerStarted","Data":"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15"} Nov 01 21:59:17.385864 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.385438 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" event={"ID":"be971324-fc2d-4b3c-b593-ebeb28f1de25","Type":"ContainerStarted","Data":"3b61915268b826dd873a4975e506fc24d2f8f5047c18f8f66a4ccc61f37d37b6"} Nov 01 21:59:17.385864 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.385596 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:17.406717 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.406667 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" podStartSLOduration=1.619826822 podStartE2EDuration="2.406654222s" totalImagesPullingTime="786.8274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:17.405339628 +0000 UTC m=+904.379242166" watchObservedRunningTime="2026-11-01 21:59:17.406654222 +0000 UTC m=+904.380556760" Nov 01 21:59:17.982514 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.982158 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 21:59:17.998219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:17.994736 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.007354 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.007324 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 21:59:18.092132 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092097 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092317 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092152 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092317 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092264 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092317 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092300 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfk85\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092494 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092473 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092513 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092566 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092560 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092619 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.092674 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.092649 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.133768 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.133740 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:18.194465 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194410 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194495 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194532 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfk85\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194692 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194665 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194733 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194809 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194866 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.194923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.194905 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.195704 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.195643 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.196016 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.195977 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.196518 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.196497 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.196660 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.196585 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.197572 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.197113 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.197572 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.197499 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.202402 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.202382 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.202972 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.202925 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.208806 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.208782 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfk85\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.211435 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.211418 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.336697 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.336661 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:18.648613 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.648587 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 21:59:18.649467 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:18.649445 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11149a0_fad0_4266_9944_0f88b9b5098e.slice/crio-c3106d99aec5fa3e4e9e2c312ac778542bf985fcc5ca7c360f76474525d063a9 WatchSource:0}: Error finding container c3106d99aec5fa3e4e9e2c312ac778542bf985fcc5ca7c360f76474525d063a9: Status 404 returned error can't find the container with id c3106d99aec5fa3e4e9e2c312ac778542bf985fcc5ca7c360f76474525d063a9 Nov 01 21:59:18.656869 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.656833 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:18.656971 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.656936 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:18.657090 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:18.656983 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:19.134127 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:19.134097 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:19.411374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:19.411284 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" event={"ID":"e11149a0-fad0-4266-9944-0f88b9b5098e","Type":"ContainerStarted","Data":"e2e2e4202294e1642b5926dec2daaa4ea84759579807a628c70ee6afedbd4fd9"} Nov 01 21:59:19.411374 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:19.411340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" event={"ID":"e11149a0-fad0-4266-9944-0f88b9b5098e","Type":"ContainerStarted","Data":"c3106d99aec5fa3e4e9e2c312ac778542bf985fcc5ca7c360f76474525d063a9"} Nov 01 21:59:19.435286 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:19.435240 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podStartSLOduration=2.435227181 podStartE2EDuration="2.435227181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.433587973 +0000 UTC m=+906.407490525" watchObservedRunningTime="2026-11-01 21:59:19.435227181 +0000 UTC m=+906.409129721" Nov 01 21:59:20.337283 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:20.337244 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:20.338683 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:20.338659 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:20.338833 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:20.338713 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:21.339102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:21.338575 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:21.339102 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:21.338646 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:22.338294 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:22.337727 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:22.338294 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:22.337793 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:22.356277 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:59:22.356237 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:59:23.337880 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:23.337840 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:23.338106 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:23.337916 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:23.423137 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:23.418972 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:24.345053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:24.340463 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:24.345053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:24.340544 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:25.337521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:25.337483 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:25.338190 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:25.338161 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:26.337901 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:26.337853 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 21:59:26.338473 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:26.337940 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 21:59:27.398353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:27.398320 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:27.400761 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:27.400736 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:27.410353 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:27.410321 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 21:59:29.944064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:29.943203 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.026058 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:30.019161 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.095274 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:30.091273 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:30.883203 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:30.883157 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 21:59:31.118984 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.118946 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.176348 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.175464 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1aee4437\"" Nov 01 21:59:31.324094 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.320636 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.324094 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.320890 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzmz\" (UniqueName: \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.427115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.423253 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzmz\" (UniqueName: \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.478055 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.460606 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.486474 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.486439 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.546061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.541434 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 21:59:31.553052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.542165 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.574056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.542827 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzmz\" (UniqueName: \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\") pod \"payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.647052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.646124 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 21:59:31.651843 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.651818 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:31.663595 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.663563 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.724145 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.713395 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1aee4437-dockercfg-6r6bs\"" Nov 01 21:59:31.743196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.724536 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:59:31.743196 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.724949 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 21:59:31.771053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.767748 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.771053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.768011 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 21:59:31.781057 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.780905 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:31.829351 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.795003 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sd6g\" (UniqueName: \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:31.829351 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.795237 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:31.908053 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.904245 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sd6g\" (UniqueName: \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:31.933566 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.933517 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:31.953176 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.953133 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:31.984858 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.984682 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.054492 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.984636 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 21:59:32.055525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:31.984490 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:32.055525 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.044918 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.406056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.401094 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 21:59:32.507340 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.507289 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:59:32.631076 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.631043 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sd6g\" (UniqueName: \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\") pod \"payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:32.697142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.674136 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.697142 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.687317 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:59:32.843063 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.837942 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 21:59:32.930229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.904367 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.930229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.916977 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:32.930229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:32.929208 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 21:59:33.014859 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.014813 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:59:33.093064 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.079894 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.135083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.095307 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.151184 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.096988 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:33.674072 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.673846 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.787083 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.773007 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:59:33.787401 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.787365 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:33.940115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.924373 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.139062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.927595 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.139062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:33.957174 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.139062 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.012660 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.154002 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.153949 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 21:59:34.154886 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.154852 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.155430 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.155405 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.291054 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.288594 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64" podUID="a389fbce-17d4-4613-884d-37aa85377992" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.524178 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.524125 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" podUID="79984a24-8785-4acc-8259-cc1e4f908fb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:59:34.591721 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.591646 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.707982 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.676592 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 21:59:34.887277 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887192 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6pk\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.887444 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887350 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.887508 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887487 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.887563 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.891751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887626 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.891751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887671 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:34.891751 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.887809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.990742 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.990822 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.990846 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.990906 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991050 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991144 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991205 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991245 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991278 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991308 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991336 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991363 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991429 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991475 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991504 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991533 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 21:59:35.012056 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:34.991561 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.044471 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.044671 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.044886 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6pk\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.045073 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.045264 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.045347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.045516 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.047012 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.057854 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.058325 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.061227 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.060756 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.081257 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.080459 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.085999 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.083138 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.093085 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.086987 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6pk\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-782nd\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.103997 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.103808 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:35.217231 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.216404 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 21:59:35.488531 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.488505 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 21:59:35.491366 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:35.491340 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39994c2_9b0f_4784_a9c6_16af42387eb3.slice/crio-cf944ea189f45fe46f6b0aa99d61751f14268d3e7cd5596f745fd672bded04d8 WatchSource:0}: Error finding container cf944ea189f45fe46f6b0aa99d61751f14268d3e7cd5596f745fd672bded04d8: Status 404 returned error can't find the container with id cf944ea189f45fe46f6b0aa99d61751f14268d3e7cd5596f745fd672bded04d8 Nov 01 21:59:35.494756 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.494722 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:35.494872 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.494790 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 21:59:35.539594 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.539567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" event={"ID":"e39994c2-9b0f-4784-a9c6-16af42387eb3","Type":"ContainerStarted","Data":"cf944ea189f45fe46f6b0aa99d61751f14268d3e7cd5596f745fd672bded04d8"} Nov 01 21:59:35.541431 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.541404 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" event={"ID":"40c12e8a-7d7d-4df5-b967-5153cfdbad94","Type":"ContainerStarted","Data":"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e"} Nov 01 21:59:35.541548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.541461 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" event={"ID":"40c12e8a-7d7d-4df5-b967-5153cfdbad94","Type":"ContainerStarted","Data":"b3a5ac8f1c02df2628a9119577434c0b60e76b9c1c939db376a40af088819622"} Nov 01 21:59:35.541548 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.541506 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:35.543149 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.543121 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" event={"ID":"9dbf6fd6-1e71-41ce-948e-ed960c70f513","Type":"ContainerStarted","Data":"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4"} Nov 01 21:59:35.543250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.543163 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" event={"ID":"9dbf6fd6-1e71-41ce-948e-ed960c70f513","Type":"ContainerStarted","Data":"85ec61a9f79d9d2186fe3af624fb0718276d80086e942ceadbe78cc58d5c4e50"} Nov 01 21:59:35.543250 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.543208 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:35.569773 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.569731 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" podStartSLOduration=5.569717796 podStartE2EDuration="5.569717796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:35.564498721 +0000 UTC m=+922.538401258" watchObservedRunningTime="2026-11-01 21:59:35.569717796 +0000 UTC m=+922.543620336" Nov 01 21:59:35.586781 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:35.586728 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" podStartSLOduration=5.586715527 podStartE2EDuration="5.586715527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:35.582875112 +0000 UTC m=+922.556777650" watchObservedRunningTime="2026-11-01 21:59:35.586715527 +0000 UTC m=+922.560618066" Nov 01 21:59:36.552433 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:36.552397 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" event={"ID":"e39994c2-9b0f-4784-a9c6-16af42387eb3","Type":"ContainerStarted","Data":"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309"} Nov 01 21:59:36.553545 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:36.553516 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:36.554449 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:36.554423 4395 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-782nd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 21:59:36.554553 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:36.554489 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 21:59:36.583897 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:36.583835 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" podStartSLOduration=4.583816785 podStartE2EDuration="4.583816785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:36.58096961 +0000 UTC m=+923.554872163" watchObservedRunningTime="2026-11-01 21:59:36.583816785 +0000 UTC m=+923.557719327" Nov 01 21:59:37.561208 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:37.561177 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 21:59:40.537320 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:40.537284 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:41.057077 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:41.053998 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:41.895708 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:41.895668 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:41.896330 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:41.895985 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" podUID="be971324-fc2d-4b3c-b593-ebeb28f1de25" containerName="maas-api" containerID="cri-o://7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15" gracePeriod=30 Nov 01 21:59:42.524080 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.511112 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:42.632052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.627520 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\") pod \"be971324-fc2d-4b3c-b593-ebeb28f1de25\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " Nov 01 21:59:42.632052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.627592 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\" (UniqueName: \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\") pod \"be971324-fc2d-4b3c-b593-ebeb28f1de25\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " Nov 01 21:59:42.632052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.627641 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\") pod \"be971324-fc2d-4b3c-b593-ebeb28f1de25\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " Nov 01 21:59:42.632052 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.628092 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\") pod \"be971324-fc2d-4b3c-b593-ebeb28f1de25\" (UID: \"be971324-fc2d-4b3c-b593-ebeb28f1de25\") " Nov 01 21:59:42.654659 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.654622 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca" pod "be971324-fc2d-4b3c-b593-ebeb28f1de25" (UID: "be971324-fc2d-4b3c-b593-ebeb28f1de25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:42.660135 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.658966 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb" pod "be971324-fc2d-4b3c-b593-ebeb28f1de25" (UID: "be971324-fc2d-4b3c-b593-ebeb28f1de25"). InnerVolumeSpecName "kube-api-access-z54vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:42.662974 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.662923 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs" pod "be971324-fc2d-4b3c-b593-ebeb28f1de25" (UID: "be971324-fc2d-4b3c-b593-ebeb28f1de25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:42.721635 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.721569 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls" pod "be971324-fc2d-4b3c-b593-ebeb28f1de25" (UID: "be971324-fc2d-4b3c-b593-ebeb28f1de25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:59:42.731400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.731231 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:42.731400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.731267 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be971324-fc2d-4b3c-b593-ebeb28f1de25-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:42.731400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.731283 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54vb\" (UniqueName: \"kubernetes.io/projected/be971324-fc2d-4b3c-b593-ebeb28f1de25-kube-api-access-z54vb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:42.731400 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.731297 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be971324-fc2d-4b3c-b593-ebeb28f1de25-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:42.859615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.859388 4395 generic.go:356] "Generic (PLEG): container finished" podID="be971324-fc2d-4b3c-b593-ebeb28f1de25" containerID="7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15" exitCode=0 Nov 01 21:59:42.859615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.859548 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" event={"ID":"be971324-fc2d-4b3c-b593-ebeb28f1de25","Type":"ContainerDied","Data":"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15"} Nov 01 21:59:42.859615 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.859613 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" event={"ID":"be971324-fc2d-4b3c-b593-ebeb28f1de25","Type":"ContainerDied","Data":"3b61915268b826dd873a4975e506fc24d2f8f5047c18f8f66a4ccc61f37d37b6"} Nov 01 21:59:42.859911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.859644 4395 scope.go:122] "RemoveContainer" containerID="7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15" Nov 01 21:59:42.859911 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.859824 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk" Nov 01 21:59:42.873270 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.873214 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:42.873888 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.873644 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" podUID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" containerName="istio-proxy" containerID="cri-o://87127a558c91895b582e1365f8693438ee8f3a68e399661e7f1e1d01a4c36493" gracePeriod=30 Nov 01 21:59:42.880787 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.880761 4395 scope.go:122] "RemoveContainer" containerID="7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15" Nov 01 21:59:42.881616 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:59:42.881593 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15\": container with ID starting with 7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15 not found: ID does not exist" containerID="7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15" Nov 01 21:59:42.882186 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.882154 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15"} err="failed to get container status \"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15\": rpc error: code = NotFound desc = could not find container \"7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15\": container with ID starting with 7b87378e34872f9b6059414eb8107657f34b2fe8b1d0222297e11ba753730e15 not found: ID does not exist" Nov 01 21:59:42.924891 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.924860 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:42.928384 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:42.927516 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93a347eb-5bf6b5b5dc-2gdkk"] Nov 01 21:59:43.252515 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:43.252477 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be971324-fc2d-4b3c-b593-ebeb28f1de25" path="/var/lib/kubelet/pods/be971324-fc2d-4b3c-b593-ebeb28f1de25/volumes" Nov 01 21:59:45.537914 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:45.537872 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" podUID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" containerName="payload-processing" containerID="cri-o://b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5" gracePeriod=25 Nov 01 21:59:45.943008 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:45.942496 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:46.009156 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.009121 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\" (UniqueName: \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\") pod \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " Nov 01 21:59:46.009513 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.009496 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\") pod \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\" (UID: \"88b5aeb6-c463-4e3a-8779-d924c51e00c0\") " Nov 01 21:59:46.009940 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.009910 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume" pod "88b5aeb6-c463-4e3a-8779-d924c51e00c0" (UID: "88b5aeb6-c463-4e3a-8779-d924c51e00c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:46.010219 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.010196 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88b5aeb6-c463-4e3a-8779-d924c51e00c0-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013158 4395 generic.go:356] "Generic (PLEG): container finished" podID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" containerID="b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5" exitCode=0 Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013227 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013248 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg" pod "88b5aeb6-c463-4e3a-8779-d924c51e00c0" (UID: "88b5aeb6-c463-4e3a-8779-d924c51e00c0"). InnerVolumeSpecName "kube-api-access-prszg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013250 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" event={"ID":"88b5aeb6-c463-4e3a-8779-d924c51e00c0","Type":"ContainerDied","Data":"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5"} Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013321 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm" event={"ID":"88b5aeb6-c463-4e3a-8779-d924c51e00c0","Type":"ContainerDied","Data":"076a6921c9c78c3cd6d8b80c7340f3a9be168c907c865eb74facff73f7dda75a"} Nov 01 21:59:46.013395 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.013356 4395 scope.go:122] "RemoveContainer" containerID="b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5" Nov 01 21:59:46.037862 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.037150 4395 scope.go:122] "RemoveContainer" containerID="b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5" Nov 01 21:59:46.038784 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:59:46.038524 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5\": container with ID starting with b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5 not found: ID does not exist" containerID="b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5" Nov 01 21:59:46.038894 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.038799 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5"} err="failed to get container status \"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5\": rpc error: code = NotFound desc = could not find container \"b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5\": container with ID starting with b157fa5dfcf62def64df3892b0d7adee8a10ccf183b84f9cfce37361d9ec65a5 not found: ID does not exist" Nov 01 21:59:46.050832 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.050805 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:46.054810 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.054789 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681db477-64946f668d-mf6dm"] Nov 01 21:59:46.054877 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.054807 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" podUID="33b1497f-cf6d-4baa-a838-06e7b276415e" containerName="payload-processing" containerID="cri-o://2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163" gracePeriod=25 Nov 01 21:59:46.111708 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.111680 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prszg\" (UniqueName: \"kubernetes.io/projected/88b5aeb6-c463-4e3a-8779-d924c51e00c0-kube-api-access-prszg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.325169 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.325108 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:46.422633 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.422595 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\") pod \"33b1497f-cf6d-4baa-a838-06e7b276415e\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " Nov 01 21:59:46.423229 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.423201 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\" (UniqueName: \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\") pod \"33b1497f-cf6d-4baa-a838-06e7b276415e\" (UID: \"33b1497f-cf6d-4baa-a838-06e7b276415e\") " Nov 01 21:59:46.427590 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.427564 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume" pod "33b1497f-cf6d-4baa-a838-06e7b276415e" (UID: "33b1497f-cf6d-4baa-a838-06e7b276415e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:46.438652 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.438589 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b1497f-cf6d-4baa-a838-06e7b276415e-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.442242 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.441963 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx" pod "33b1497f-cf6d-4baa-a838-06e7b276415e" (UID: "33b1497f-cf6d-4baa-a838-06e7b276415e"). InnerVolumeSpecName "kube-api-access-nddsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:46.565870 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.565820 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddsx\" (UniqueName: \"kubernetes.io/projected/33b1497f-cf6d-4baa-a838-06e7b276415e-kube-api-access-nddsx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:46.569516 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.569488 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 21:59:46.569701 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:46.569681 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 21:59:47.023734 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.023704 4395 generic.go:356] "Generic (PLEG): container finished" podID="33b1497f-cf6d-4baa-a838-06e7b276415e" containerID="2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163" exitCode=0 Nov 01 21:59:47.023923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.023746 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" Nov 01 21:59:47.023923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.023798 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" event={"ID":"33b1497f-cf6d-4baa-a838-06e7b276415e","Type":"ContainerDied","Data":"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163"} Nov 01 21:59:47.023923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.023844 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv" event={"ID":"33b1497f-cf6d-4baa-a838-06e7b276415e","Type":"ContainerDied","Data":"9cf146b0ae63787f1c08a0bef1dc16438a73e96042d4355805e9ca70c3fe0949"} Nov 01 21:59:47.023923 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.023867 4395 scope.go:122] "RemoveContainer" containerID="2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163" Nov 01 21:59:47.034722 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.034700 4395 scope.go:122] "RemoveContainer" containerID="2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163" Nov 01 21:59:47.035006 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:59:47.034988 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163\": container with ID starting with 2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163 not found: ID does not exist" containerID="2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163" Nov 01 21:59:47.035113 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.035012 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163"} err="failed to get container status \"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163\": rpc error: code = NotFound desc = could not find container \"2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163\": container with ID starting with 2fe793f27ef2f8ea2bf3de0382328e9fe44d3d15650067102d78ac0f51a0a163 not found: ID does not exist" Nov 01 21:59:47.049393 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.049373 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:47.056630 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.056603 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-13364262-8f8d44b44-ndwvv"] Nov 01 21:59:47.253899 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.253864 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b1497f-cf6d-4baa-a838-06e7b276415e" path="/var/lib/kubelet/pods/33b1497f-cf6d-4baa-a838-06e7b276415e/volumes" Nov 01 21:59:47.254403 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:47.254388 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" path="/var/lib/kubelet/pods/88b5aeb6-c463-4e3a-8779-d924c51e00c0/volumes" Nov 01 21:59:48.035934 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.035895 4395 generic.go:356] "Generic (PLEG): container finished" podID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" containerID="87127a558c91895b582e1365f8693438ee8f3a68e399661e7f1e1d01a4c36493" exitCode=0 Nov 01 21:59:48.036422 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.035975 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" event={"ID":"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008","Type":"ContainerDied","Data":"87127a558c91895b582e1365f8693438ee8f3a68e399661e7f1e1d01a4c36493"} Nov 01 21:59:48.157852 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.157774 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:48.316071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316004 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316071 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316062 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316200 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316236 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316263 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316308 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316293 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316348 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316445 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316521 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316502 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\") pod \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\" (UID: \"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008\") " Nov 01 21:59:48.316679 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316582 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:48.316730 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:48.316991 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.316966 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:48.317148 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317124 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:48.317336 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317312 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:59:48.317509 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317488 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.317581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317517 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.317581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317533 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.317581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317551 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.317581 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.317565 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.318688 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.318655 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "kube-api-access-87gmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:48.318688 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.318661 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:59:48.319993 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.319973 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:59:48.320075 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.320004 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo" pod "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" (UID: "7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:59:48.419469 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.419406 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87gmh\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-kube-api-access-87gmh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.419469 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.419431 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.419469 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.419441 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:48.419469 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:48.419451 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 21:59:49.057766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.057726 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" Nov 01 21:59:49.057766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.057751 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j" event={"ID":"7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008","Type":"ContainerDied","Data":"f1081ddb0f2e0e3bc7c80399aa43f1dd894df45ae77e915a106b1168fc36b3a3"} Nov 01 21:59:49.058319 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.057827 4395 scope.go:122] "RemoveContainer" containerID="87127a558c91895b582e1365f8693438ee8f3a68e399661e7f1e1d01a4c36493" Nov 01 21:59:49.100241 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.100214 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:49.107648 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.107624 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-93a347eb-openshift-default-95778769c-7g26j"] Nov 01 21:59:49.171289 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:49.171262 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" path="/var/lib/kubelet/pods/7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008/volumes" Nov 01 21:59:52.302045 ip-10-0-136-47 kubenswrapper[4395]: E1101 21:59:52.301995 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 21:59:55.334115 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334016 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 21:59:55.334879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334860 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" containerName="istio-proxy" Nov 01 21:59:55.334879 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334880 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" containerName="istio-proxy" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334897 4395 cpu_manager.go:414] "containerMap: removing container" podUID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" containerName="payload-processing" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334903 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" containerName="payload-processing" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334921 4395 cpu_manager.go:414] "containerMap: removing container" podUID="33b1497f-cf6d-4baa-a838-06e7b276415e" containerName="payload-processing" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334929 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1497f-cf6d-4baa-a838-06e7b276415e" containerName="payload-processing" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334957 4395 cpu_manager.go:414] "containerMap: removing container" podUID="be971324-fc2d-4b3c-b593-ebeb28f1de25" containerName="maas-api" Nov 01 21:59:55.334973 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.334961 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be971324-fc2d-4b3c-b593-ebeb28f1de25" containerName="maas-api" Nov 01 21:59:55.335222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.335069 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7337c1f2-8f6f-4a7a-9f8f-b63b6cfb8008" containerName="istio-proxy" Nov 01 21:59:55.335222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.335078 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b5aeb6-c463-4e3a-8779-d924c51e00c0" containerName="payload-processing" Nov 01 21:59:55.335222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.335088 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1497f-cf6d-4baa-a838-06e7b276415e" containerName="payload-processing" Nov 01 21:59:55.335222 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.335098 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="be971324-fc2d-4b3c-b593-ebeb28f1de25" containerName="maas-api" Nov 01 21:59:55.341164 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.341142 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.347508 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.347478 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d5713372\"" Nov 01 21:59:55.348483 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.348459 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d5713372-dockercfg-gfg28\"" Nov 01 21:59:55.348995 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.348971 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 21:59:55.367128 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.367097 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.367128 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.367130 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkxw\" (UniqueName: \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.399458 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.399423 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 21:59:55.404636 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.404608 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.426451 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.426427 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 21:59:55.706104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.696288 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.706104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.696331 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkxw\" (UniqueName: \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.706104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.696425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.706104 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.696608 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkjm\" (UniqueName: \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.716061 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.714432 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.764593 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.764555 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkxw\" (UniqueName: \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\") pod \"payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:55.800093 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.798318 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkjm\" (UniqueName: \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.800093 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.798756 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.800504 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.800463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.814968 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.814931 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkjm\" (UniqueName: \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\") pod \"payload-processing-e2e-derive-d5713372-5555895f6-lk42f\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:55.958423 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:55.958340 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:56.039382 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:56.039339 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:56.250597 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:56.250565 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 21:59:56.250924 ip-10-0-136-47 kubenswrapper[4395]: W1101 21:59:56.250893 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bb7c59_7b35_4c04_b920_dda37b605c05.slice/crio-45d1b1c2a6090dc8ffde734043a035522faa214970a9faf7bbbfd1dd74588ddf WatchSource:0}: Error finding container 45d1b1c2a6090dc8ffde734043a035522faa214970a9faf7bbbfd1dd74588ddf: Status 404 returned error can't find the container with id 45d1b1c2a6090dc8ffde734043a035522faa214970a9faf7bbbfd1dd74588ddf Nov 01 21:59:56.377609 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:56.377574 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 21:59:57.274095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.265416 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 21:59:57.274095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.265480 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" event={"ID":"c9c74556-6993-48dd-af77-3f78274db7fb","Type":"ContainerStarted","Data":"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733"} Nov 01 21:59:57.274095 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.265590 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" event={"ID":"c9c74556-6993-48dd-af77-3f78274db7fb","Type":"ContainerStarted","Data":"672e330d037396aa70f23b6700c9ecf905fe550d94dcfd92200b01c1140e04b2"} Nov 01 21:59:57.295065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.292466 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" event={"ID":"25bb7c59-7b35-4c04-b920-dda37b605c05","Type":"ContainerStarted","Data":"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417"} Nov 01 21:59:57.295065 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.292627 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" event={"ID":"25bb7c59-7b35-4c04-b920-dda37b605c05","Type":"ContainerStarted","Data":"45d1b1c2a6090dc8ffde734043a035522faa214970a9faf7bbbfd1dd74588ddf"} Nov 01 21:59:57.310473 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.308994 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 21:59:57.312935 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.312845 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" podStartSLOduration=2.312823384 podStartE2EDuration="2.312823384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:57.304562392 +0000 UTC m=+944.278464943" watchObservedRunningTime="2026-11-01 21:59:57.312823384 +0000 UTC m=+944.286725924" Nov 01 21:59:57.329766 ip-10-0-136-47 kubenswrapper[4395]: I1101 21:59:57.329712 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" podStartSLOduration=2.329697165 podStartE2EDuration="2.329697165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:57.326977424 +0000 UTC m=+944.300879963" watchObservedRunningTime="2026-11-01 21:59:57.329697165 +0000 UTC m=+944.303599705" Nov 01 22:00:00.156240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.156121 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:00.161962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.161810 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:00.161962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.161890 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.180700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.180643 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.180919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.180895 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:00:00.206320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.206284 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:00.220391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.220359 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:00.230118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.230088 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:00.241249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.241172 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:00.241249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.241237 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.335012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.334949 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2mw\" (UniqueName: \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.335928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.335898 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdpg\" (UniqueName: \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.336306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.336286 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.346131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.346094 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.449053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.448952 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmxf\" (UniqueName: \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.449202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449018 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2mw\" (UniqueName: \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.449202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449124 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.449276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449233 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdpg\" (UniqueName: \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.449345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449333 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.449424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449405 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.449934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449917 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.449934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.449929 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.462326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.462298 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2mw\" (UniqueName: \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.462501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.462333 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdpg\" (UniqueName: \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.538089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.538060 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:00.548359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.548333 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:00.550249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.550187 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmxf\" (UniqueName: \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.550316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.550249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.551300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.551272 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.563339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.563314 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmxf\" (UniqueName: \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:00.841498 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:00:00.841463 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94731ae6_1efa_4e04_8848_92bcf6a5f771.slice/crio-4e7df45b79a7e7050f89d77a28f4d9226e47d37469daae70b3164de30eda971a WatchSource:0}: Error finding container 4e7df45b79a7e7050f89d77a28f4d9226e47d37469daae70b3164de30eda971a: Status 404 returned error can't find the container with id 4e7df45b79a7e7050f89d77a28f4d9226e47d37469daae70b3164de30eda971a Nov 01 22:00:00.841647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.841514 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:00.845479 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:00:00.845456 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ed42dd_bf96_4996_a67d_b87fc9d6d38e.slice/crio-94a14de40a4df9832dce10f0e8ae1a89bf1bbc516ab0673b74e175417b759ada WatchSource:0}: Error finding container 94a14de40a4df9832dce10f0e8ae1a89bf1bbc516ab0673b74e175417b759ada: Status 404 returned error can't find the container with id 94a14de40a4df9832dce10f0e8ae1a89bf1bbc516ab0673b74e175417b759ada Nov 01 22:00:00.846154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.846130 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:00.859876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:00.859856 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:01.229455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.229410 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:01.374759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.374722 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerStarted","Data":"91a3980d30de969645ec817797eca49e7f3cfd269109835fef18b96f6a20db94"} Nov 01 22:00:01.374921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.374799 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerStarted","Data":"94a14de40a4df9832dce10f0e8ae1a89bf1bbc516ab0673b74e175417b759ada"} Nov 01 22:00:01.376918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.376892 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerStarted","Data":"68860d55fb87642b38387433b435f0637a9512bb253c4d5a3c64097d5358c569"} Nov 01 22:00:01.378539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.378511 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerStarted","Data":"4e7df45b79a7e7050f89d77a28f4d9226e47d37469daae70b3164de30eda971a"} Nov 01 22:00:01.395992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:01.395949 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" podStartSLOduration=1.39593648 podStartE2EDuration="1.39593648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:01.393438182 +0000 UTC m=+948.367340725" watchObservedRunningTime="2026-11-01 22:00:01.39593648 +0000 UTC m=+948.369839061" Nov 01 22:00:02.387812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.387778 4395 generic.go:356] "Generic (PLEG): container finished" podID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerID="91a3980d30de969645ec817797eca49e7f3cfd269109835fef18b96f6a20db94" exitCode=60 Nov 01 22:00:02.388188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.387866 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerDied","Data":"91a3980d30de969645ec817797eca49e7f3cfd269109835fef18b96f6a20db94"} Nov 01 22:00:02.388225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.388206 4395 scope.go:122] "RemoveContainer" containerID="91a3980d30de969645ec817797eca49e7f3cfd269109835fef18b96f6a20db94" Nov 01 22:00:02.389540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.389521 4395 generic.go:356] "Generic (PLEG): container finished" podID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerID="6435ccda8169dcbb28d4ce6481204de8afa9c9bf6861bfdbca3e479f257f2794" exitCode=60 Nov 01 22:00:02.389643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.389542 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerDied","Data":"6435ccda8169dcbb28d4ce6481204de8afa9c9bf6861bfdbca3e479f257f2794"} Nov 01 22:00:02.389862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.389846 4395 scope.go:122] "RemoveContainer" containerID="6435ccda8169dcbb28d4ce6481204de8afa9c9bf6861bfdbca3e479f257f2794" Nov 01 22:00:02.391484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.391456 4395 generic.go:356] "Generic (PLEG): container finished" podID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerID="0573184af0ea58a5747bf325ed1c47cafd07be5aea6d7ee8b08e164ba4562324" exitCode=60 Nov 01 22:00:02.391571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.391520 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerDied","Data":"0573184af0ea58a5747bf325ed1c47cafd07be5aea6d7ee8b08e164ba4562324"} Nov 01 22:00:02.391760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:02.391722 4395 scope.go:122] "RemoveContainer" containerID="0573184af0ea58a5747bf325ed1c47cafd07be5aea6d7ee8b08e164ba4562324" Nov 01 22:00:03.403362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:03.403329 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerStarted","Data":"776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724"} Nov 01 22:00:03.405510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:03.405483 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerStarted","Data":"71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b"} Nov 01 22:00:03.407605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:03.407582 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerStarted","Data":"ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef"} Nov 01 22:00:03.440334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:03.440285 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" podStartSLOduration=3.440272804 podStartE2EDuration="3.440272804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:03.437134414 +0000 UTC m=+950.411036978" watchObservedRunningTime="2026-11-01 22:00:03.440272804 +0000 UTC m=+950.414175343" Nov 01 22:00:03.456957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:03.456911 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" podStartSLOduration=3.456899174 podStartE2EDuration="3.456899174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:03.454733608 +0000 UTC m=+950.428636171" watchObservedRunningTime="2026-11-01 22:00:03.456899174 +0000 UTC m=+950.430801713" Nov 01 22:00:04.419142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.419113 4395 generic.go:356] "Generic (PLEG): container finished" podID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerID="776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724" exitCode=60 Nov 01 22:00:04.419700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.419197 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerDied","Data":"776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724"} Nov 01 22:00:04.419700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.419266 4395 scope.go:122] "RemoveContainer" containerID="91a3980d30de969645ec817797eca49e7f3cfd269109835fef18b96f6a20db94" Nov 01 22:00:04.419700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.419453 4395 scope.go:122] "RemoveContainer" containerID="776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724" Nov 01 22:00:04.419837 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:04.419745 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr_odh-ai-gateway-infra(b1ed42dd-bf96-4996-a67d-b87fc9d6d38e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" Nov 01 22:00:04.421424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.421398 4395 generic.go:356] "Generic (PLEG): container finished" podID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerID="71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b" exitCode=60 Nov 01 22:00:04.421548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.421531 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerDied","Data":"71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b"} Nov 01 22:00:04.421706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.421692 4395 scope.go:122] "RemoveContainer" containerID="71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b" Nov 01 22:00:04.422009 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:04.421986 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb_odh-ai-gateway-infra(fba0e63c-50d1-492c-86aa-2a5ba04a0e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" Nov 01 22:00:04.423574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.423549 4395 generic.go:356] "Generic (PLEG): container finished" podID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerID="ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef" exitCode=60 Nov 01 22:00:04.423678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.423631 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerDied","Data":"ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef"} Nov 01 22:00:04.423815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.423798 4395 scope.go:122] "RemoveContainer" containerID="ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef" Nov 01 22:00:04.424020 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:04.424004 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7_odh-ai-gateway-infra(94731ae6-1efa-4e04-8848-92bcf6a5f771)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" Nov 01 22:00:04.432621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.432607 4395 scope.go:122] "RemoveContainer" containerID="6435ccda8169dcbb28d4ce6481204de8afa9c9bf6861bfdbca3e479f257f2794" Nov 01 22:00:04.446169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:04.446126 4395 scope.go:122] "RemoveContainer" containerID="0573184af0ea58a5747bf325ed1c47cafd07be5aea6d7ee8b08e164ba4562324" Nov 01 22:00:05.434512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:05.434473 4395 scope.go:122] "RemoveContainer" containerID="776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724" Nov 01 22:00:05.434979 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:05.434865 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr_odh-ai-gateway-infra(b1ed42dd-bf96-4996-a67d-b87fc9d6d38e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" Nov 01 22:00:05.436502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:05.436475 4395 scope.go:122] "RemoveContainer" containerID="71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b" Nov 01 22:00:05.436866 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:05.436840 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb_odh-ai-gateway-infra(fba0e63c-50d1-492c-86aa-2a5ba04a0e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" Nov 01 22:00:05.438513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:05.438491 4395 scope.go:122] "RemoveContainer" containerID="ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef" Nov 01 22:00:05.438951 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:05.438908 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7_odh-ai-gateway-infra(94731ae6-1efa-4e04-8848-92bcf6a5f771)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" Nov 01 22:00:07.680334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.680283 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.715191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.715144 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.764537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.764495 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.936249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.936137 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.973058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.958155 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:07.973058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:07.959273 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.048797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.047680 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.464320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.464291 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 22:00:08.498408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.497114 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.716569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.716469 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:00:08.717598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.717575 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:00:08.776102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.771193 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:08.792162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:08.792112 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:00:09.089072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.077204 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.089072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.078448 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:00:09.089072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.080477 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:00:09.089072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.080542 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:00:09.114543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.110163 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:00:09.114543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.111128 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:00:09.114543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.111790 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:00:09.114543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.111870 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:00:09.504236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.504204 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.513055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.506944 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:00:09.672098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.672054 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:00:09.736711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.736673 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 01 22:00:09.793110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.792971 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:09.794203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.794174 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:00:09.847308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:09.847271 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 22:00:10.078103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.077985 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.142054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.078955 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.142054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.132984 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:00:10.142054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.133021 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:00:10.142353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.133074 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:00:10.509653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.509603 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:10.703058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.682245 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:00:10.840126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.837005 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:00:10.949706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.949652 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.061097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:10.951555 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:00:11.061097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.053338 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.093420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.093323 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 01 22:00:11.195063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.183341 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:00:11.195063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.185065 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 22:00:11.197052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.196994 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:00:11.219092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.210194 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:00:11.595053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.567185 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.761053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.734913 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:00:11.810089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.778199 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:11.912296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.912197 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.007152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:11.984744 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.084637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.040136 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.092679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.081062 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:00:12.092679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.092328 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.093755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.093679 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.259174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.250367 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:00:12.259174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.253751 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 22:00:12.259174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.256021 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.276052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.272656 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.500213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.500161 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 22:00:12.729514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.728993 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 22:00:12.748955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.748586 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.779862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.779793 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:12.846574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.846542 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:00:12.914166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:12.914123 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.104065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.094154 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.256050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.250662 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:00:13.269475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.256342 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.273935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.273897 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.309227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.292201 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:13.520774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.520747 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:00:13.520993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.520981 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:00:13.536663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.521726 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:00:13.536663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.522584 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:00:13.545695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:13.545673 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:00:14.133878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:14.133822 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:00:14.164727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:14.164660 4395 scope.go:122] "RemoveContainer" containerID="776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724" Nov 01 22:00:14.168385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:14.168362 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:00:14.257758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:14.251455 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:00:15.079698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:15.079659 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerStarted","Data":"13d7cea57498bfc27c8665a496e91880b4553e53779baebc752a5500423b626b"} Nov 01 22:00:15.159505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:15.159468 4395 scope.go:122] "RemoveContainer" containerID="71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b" Nov 01 22:00:15.159890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:15.159634 4395 scope.go:122] "RemoveContainer" containerID="ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef" Nov 01 22:00:16.089534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.089503 4395 generic.go:356] "Generic (PLEG): container finished" podID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerID="13d7cea57498bfc27c8665a496e91880b4553e53779baebc752a5500423b626b" exitCode=60 Nov 01 22:00:16.089697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.089581 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerDied","Data":"13d7cea57498bfc27c8665a496e91880b4553e53779baebc752a5500423b626b"} Nov 01 22:00:16.089697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.089641 4395 scope.go:122] "RemoveContainer" containerID="776f5bae1f6679e553a20db1a6ae656ef3839cb5f2892c76bbd4cee9f6279724" Nov 01 22:00:16.090095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.090070 4395 scope.go:122] "RemoveContainer" containerID="13d7cea57498bfc27c8665a496e91880b4553e53779baebc752a5500423b626b" Nov 01 22:00:16.090447 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:16.090426 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr_odh-ai-gateway-infra(b1ed42dd-bf96-4996-a67d-b87fc9d6d38e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" Nov 01 22:00:16.091598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.091576 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerStarted","Data":"02207007c940fd43452e0d2a1cc2dabcc9969d4dbd167e6d0bf5477bf724879c"} Nov 01 22:00:16.093411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.093394 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerStarted","Data":"69b62304416486b6b6ca56f69a4eeab78e53850a66fc6f3fca7522bb78c2e6da"} Nov 01 22:00:16.109972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:16.109947 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:17.512761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.512681 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" podUID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" containerName="payload-processing" containerID="cri-o://d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e" gracePeriod=25 Nov 01 22:00:17.581928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.581897 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:17.582185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.582158 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:17.587414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.587386 4395 generic.go:356] "Generic (PLEG): container finished" podID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerID="02207007c940fd43452e0d2a1cc2dabcc9969d4dbd167e6d0bf5477bf724879c" exitCode=60 Nov 01 22:00:17.589155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.588977 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerDied","Data":"02207007c940fd43452e0d2a1cc2dabcc9969d4dbd167e6d0bf5477bf724879c"} Nov 01 22:00:17.589374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.589322 4395 scope.go:122] "RemoveContainer" containerID="71f4f4d4d73322a95c4b3d7d4a15bef4889c1b620bd830a5548f64e5047ca41b" Nov 01 22:00:17.594596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.594569 4395 generic.go:356] "Generic (PLEG): container finished" podID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerID="69b62304416486b6b6ca56f69a4eeab78e53850a66fc6f3fca7522bb78c2e6da" exitCode=60 Nov 01 22:00:17.594870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.594847 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerDied","Data":"69b62304416486b6b6ca56f69a4eeab78e53850a66fc6f3fca7522bb78c2e6da"} Nov 01 22:00:17.676175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.675994 4395 scope.go:122] "RemoveContainer" containerID="ecc87c9f6f89ddc6cf9c390653bceed72ece5b393951b6883c49b01592c515ef" Nov 01 22:00:17.731245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.731097 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" podUID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" containerName="payload-pre-processing" containerID="cri-o://ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4" gracePeriod=25 Nov 01 22:00:17.892228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.890633 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:17.904716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.904693 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:17.928072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.928010 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:17.955543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.955500 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\" (UniqueName: \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\") pod \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " Nov 01 22:00:17.955704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.955685 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\") pod \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\" (UID: \"fba0e63c-50d1-492c-86aa-2a5ba04a0e90\") " Nov 01 22:00:17.955804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.955786 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\") pod \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " Nov 01 22:00:17.956646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.955885 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\" (UniqueName: \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\") pod \"94731ae6-1efa-4e04-8848-92bcf6a5f771\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " Nov 01 22:00:17.956646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.956185 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\" (UniqueName: \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\") pod \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\" (UID: \"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e\") " Nov 01 22:00:17.956646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.956291 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca" pod "b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" (UID: "b1ed42dd-bf96-4996-a67d-b87fc9d6d38e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:17.956646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.956301 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\") pod \"94731ae6-1efa-4e04-8848-92bcf6a5f771\" (UID: \"94731ae6-1efa-4e04-8848-92bcf6a5f771\") " Nov 01 22:00:17.956646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.956614 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca" pod "94731ae6-1efa-4e04-8848-92bcf6a5f771" (UID: "94731ae6-1efa-4e04-8848-92bcf6a5f771"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:17.957241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.957147 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:17.957241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.957171 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94731ae6-1efa-4e04-8848-92bcf6a5f771-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:17.957548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.957506 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca" pod "fba0e63c-50d1-492c-86aa-2a5ba04a0e90" (UID: "fba0e63c-50d1-492c-86aa-2a5ba04a0e90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:17.959661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.959633 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw" pod "b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" (UID: "b1ed42dd-bf96-4996-a67d-b87fc9d6d38e"). InnerVolumeSpecName "kube-api-access-fx2mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:17.969439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.966308 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf" pod "fba0e63c-50d1-492c-86aa-2a5ba04a0e90" (UID: "fba0e63c-50d1-492c-86aa-2a5ba04a0e90"). InnerVolumeSpecName "kube-api-access-kfmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:17.970279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:17.970251 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg" pod "94731ae6-1efa-4e04-8848-92bcf6a5f771" (UID: "94731ae6-1efa-4e04-8848-92bcf6a5f771"). InnerVolumeSpecName "kube-api-access-kqdpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:18.030669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.030628 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 22:00:18.058628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.057922 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\") pod \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " Nov 01 22:00:18.058628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.057997 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\" (UniqueName: \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\") pod \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\" (UID: \"40c12e8a-7d7d-4df5-b967-5153cfdbad94\") " Nov 01 22:00:18.058628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058483 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume" pod "40c12e8a-7d7d-4df5-b967-5153cfdbad94" (UID: "40c12e8a-7d7d-4df5-b967-5153cfdbad94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:18.058896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058820 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c12e8a-7d7d-4df5-b967-5153cfdbad94-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.058896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058838 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmxf\" (UniqueName: \"kubernetes.io/projected/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-kube-api-access-kfmxf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.058896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058848 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba0e63c-50d1-492c-86aa-2a5ba04a0e90-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.058896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058856 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqdpg\" (UniqueName: \"kubernetes.io/projected/94731ae6-1efa-4e04-8848-92bcf6a5f771-kube-api-access-kqdpg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.058896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.058867 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx2mw\" (UniqueName: \"kubernetes.io/projected/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e-kube-api-access-fx2mw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.060875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.060847 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g" pod "40c12e8a-7d7d-4df5-b967-5153cfdbad94" (UID: "40c12e8a-7d7d-4df5-b967-5153cfdbad94"). InnerVolumeSpecName "kube-api-access-4sd6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:18.094293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.094239 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 22:00:18.159537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.159502 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\" (UniqueName: \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\") pod \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " Nov 01 22:00:18.159790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.159770 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\") pod \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\" (UID: \"9dbf6fd6-1e71-41ce-948e-ed960c70f513\") " Nov 01 22:00:18.160217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.160180 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume" pod "9dbf6fd6-1e71-41ce-948e-ed960c70f513" (UID: "9dbf6fd6-1e71-41ce-948e-ed960c70f513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:18.161110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.160908 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dbf6fd6-1e71-41ce-948e-ed960c70f513-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.161110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.160939 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sd6g\" (UniqueName: \"kubernetes.io/projected/40c12e8a-7d7d-4df5-b967-5153cfdbad94-kube-api-access-4sd6g\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.162462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.162423 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz" pod "9dbf6fd6-1e71-41ce-948e-ed960c70f513" (UID: "9dbf6fd6-1e71-41ce-948e-ed960c70f513"). InnerVolumeSpecName "kube-api-access-zwzmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:18.262905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.262873 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzmz\" (UniqueName: \"kubernetes.io/projected/9dbf6fd6-1e71-41ce-948e-ed960c70f513-kube-api-access-zwzmz\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:18.604006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.603972 4395 generic.go:356] "Generic (PLEG): container finished" podID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" containerID="d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e" exitCode=0 Nov 01 22:00:18.604492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.604021 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" Nov 01 22:00:18.604492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.604057 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" event={"ID":"40c12e8a-7d7d-4df5-b967-5153cfdbad94","Type":"ContainerDied","Data":"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e"} Nov 01 22:00:18.604492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.604185 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm" event={"ID":"40c12e8a-7d7d-4df5-b967-5153cfdbad94","Type":"ContainerDied","Data":"b3a5ac8f1c02df2628a9119577434c0b60e76b9c1c939db376a40af088819622"} Nov 01 22:00:18.604492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.604215 4395 scope.go:122] "RemoveContainer" containerID="d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e" Nov 01 22:00:18.605707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.605689 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" Nov 01 22:00:18.605797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.605698 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr" event={"ID":"b1ed42dd-bf96-4996-a67d-b87fc9d6d38e","Type":"ContainerDied","Data":"94a14de40a4df9832dce10f0e8ae1a89bf1bbc516ab0673b74e175417b759ada"} Nov 01 22:00:18.607320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.607292 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" Nov 01 22:00:18.607448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.607323 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb" event={"ID":"fba0e63c-50d1-492c-86aa-2a5ba04a0e90","Type":"ContainerDied","Data":"68860d55fb87642b38387433b435f0637a9512bb253c4d5a3c64097d5358c569"} Nov 01 22:00:18.609333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.609310 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" Nov 01 22:00:18.609508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.609376 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7" event={"ID":"94731ae6-1efa-4e04-8848-92bcf6a5f771","Type":"ContainerDied","Data":"4e7df45b79a7e7050f89d77a28f4d9226e47d37469daae70b3164de30eda971a"} Nov 01 22:00:18.610978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.610952 4395 generic.go:356] "Generic (PLEG): container finished" podID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" containerID="ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4" exitCode=0 Nov 01 22:00:18.611126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.610983 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" Nov 01 22:00:18.611126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.611007 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" event={"ID":"9dbf6fd6-1e71-41ce-948e-ed960c70f513","Type":"ContainerDied","Data":"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4"} Nov 01 22:00:18.611126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.611061 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6" event={"ID":"9dbf6fd6-1e71-41ce-948e-ed960c70f513","Type":"ContainerDied","Data":"85ec61a9f79d9d2186fe3af624fb0718276d80086e942ceadbe78cc58d5c4e50"} Nov 01 22:00:18.648223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.648203 4395 scope.go:122] "RemoveContainer" containerID="d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e" Nov 01 22:00:18.648503 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:18.648481 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e\": container with ID starting with d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e not found: ID does not exist" containerID="d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e" Nov 01 22:00:18.648570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.648514 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e"} err="failed to get container status \"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e\": rpc error: code = NotFound desc = could not find container \"d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e\": container with ID starting with d97451156767ed266f79020eed89e6a3d9990446b00269037a5e7fd5a507661e not found: ID does not exist" Nov 01 22:00:18.648570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.648537 4395 scope.go:122] "RemoveContainer" containerID="13d7cea57498bfc27c8665a496e91880b4553e53779baebc752a5500423b626b" Nov 01 22:00:18.658897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.658878 4395 scope.go:122] "RemoveContainer" containerID="02207007c940fd43452e0d2a1cc2dabcc9969d4dbd167e6d0bf5477bf724879c" Nov 01 22:00:18.667811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.667791 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 22:00:18.669112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.669088 4395 scope.go:122] "RemoveContainer" containerID="69b62304416486b6b6ca56f69a4eeab78e53850a66fc6f3fca7522bb78c2e6da" Nov 01 22:00:18.672553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.672534 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1aee4437-8649b8584d-mq9rm"] Nov 01 22:00:18.680155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.680131 4395 scope.go:122] "RemoveContainer" containerID="ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4" Nov 01 22:00:18.688568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.688540 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 22:00:18.695068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.695040 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1aee4437-867f875c6c-vdcf6"] Nov 01 22:00:18.703265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.703243 4395 scope.go:122] "RemoveContainer" containerID="ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4" Nov 01 22:00:18.703527 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:18.703509 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4\": container with ID starting with ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4 not found: ID does not exist" containerID="ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4" Nov 01 22:00:18.703579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.703536 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4"} err="failed to get container status \"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4\": rpc error: code = NotFound desc = could not find container \"ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4\": container with ID starting with ff2f0a4e4844ec1fb036a7765337dbee416b55e30e09584d4531a0f903d35bd4 not found: ID does not exist" Nov 01 22:00:18.703960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.703944 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:18.713743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.713717 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892840-pvfvb"] Nov 01 22:00:18.724934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.724912 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:18.729095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.729067 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fb9597-29892840-b8dv7"] Nov 01 22:00:18.742595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.742572 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:18.752896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:18.752875 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892840-624vr"] Nov 01 22:00:19.168751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:19.168718 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" path="/var/lib/kubelet/pods/40c12e8a-7d7d-4df5-b967-5153cfdbad94/volumes" Nov 01 22:00:19.169301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:19.169278 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" path="/var/lib/kubelet/pods/94731ae6-1efa-4e04-8848-92bcf6a5f771/volumes" Nov 01 22:00:19.169847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:19.169812 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" path="/var/lib/kubelet/pods/9dbf6fd6-1e71-41ce-948e-ed960c70f513/volumes" Nov 01 22:00:19.170584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:19.170567 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" path="/var/lib/kubelet/pods/b1ed42dd-bf96-4996-a67d-b87fc9d6d38e/volumes" Nov 01 22:00:19.170980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:19.170966 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" path="/var/lib/kubelet/pods/fba0e63c-50d1-492c-86aa-2a5ba04a0e90/volumes" Nov 01 22:00:22.311669 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:22.311623 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:00:29.288758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:29.288708 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 22:00:29.302664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:29.302631 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 22:00:30.580956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.580923 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:00:30.582185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582136 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.582185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582163 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582193 4395 cpu_manager.go:414] "containerMap: removing container" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582201 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582211 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582218 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582233 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582240 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582260 4395 cpu_manager.go:414] "containerMap: removing container" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582267 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582278 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582283 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582293 4395 cpu_manager.go:414] "containerMap: removing container" podUID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" containerName="payload-processing" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582297 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" containerName="payload-processing" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582332 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" containerName="payload-pre-processing" Nov 01 22:00:30.582377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582336 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" containerName="payload-pre-processing" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582491 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582505 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582514 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582524 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c12e8a-7d7d-4df5-b967-5153cfdbad94" containerName="payload-processing" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582536 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582543 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582554 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582563 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbf6fd6-1e71-41ce-948e-ed960c70f513" containerName="payload-pre-processing" Nov 01 22:00:30.583078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.582574 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:30.586541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.586517 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.595070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.594997 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-1783e3-dockercfg-zkdcv\"" Nov 01 22:00:30.601813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.601793 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:00:30.640307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.640271 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.640539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.640352 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzwx\" (UniqueName: \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.682014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.681854 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1783e3\"" Nov 01 22:00:30.750057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.744399 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.750057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.744508 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzwx\" (UniqueName: \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.750057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.745864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.791760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.778968 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzwx\" (UniqueName: \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\") pod \"payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:30.928145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:30.926335 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:31.393639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:31.393596 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:00:31.893641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:31.893604 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" event={"ID":"13131b61-d3c7-4702-9d42-32469767084f","Type":"ContainerStarted","Data":"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658"} Nov 01 22:00:31.894066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:31.893667 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" event={"ID":"13131b61-d3c7-4702-9d42-32469767084f","Type":"ContainerStarted","Data":"ddbe827858849a9bb90d2598b6ae1521b199c5d434fe033557511a2f251c2863"} Nov 01 22:00:31.894066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:31.893690 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:31.913426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:31.913374 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" podStartSLOduration=1.9133612100000001 podStartE2EDuration="1.91336121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:31.910682204 +0000 UTC m=+978.884584746" watchObservedRunningTime="2026-11-01 22:00:31.91336121 +0000 UTC m=+978.887263748" Nov 01 22:00:34.289822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.289788 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" podUID="c9c74556-6993-48dd-af77-3f78274db7fb" containerName="payload-processing" containerID="cri-o://d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733" gracePeriod=25 Nov 01 22:00:34.303394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.303363 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" podUID="25bb7c59-7b35-4c04-b920-dda37b605c05" containerName="payload-pre-processing" containerID="cri-o://d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417" gracePeriod=25 Nov 01 22:00:34.709564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.709533 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 22:00:34.716602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.716577 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\") pod \"c9c74556-6993-48dd-af77-3f78274db7fb\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " Nov 01 22:00:34.716747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.716739 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\" (UniqueName: \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\") pod \"c9c74556-6993-48dd-af77-3f78274db7fb\" (UID: \"c9c74556-6993-48dd-af77-3f78274db7fb\") " Nov 01 22:00:34.716912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.716893 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume" pod "c9c74556-6993-48dd-af77-3f78274db7fb" (UID: "c9c74556-6993-48dd-af77-3f78274db7fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:34.717348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.717332 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74556-6993-48dd-af77-3f78274db7fb-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:34.722499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.722471 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm" pod "c9c74556-6993-48dd-af77-3f78274db7fb" (UID: "c9c74556-6993-48dd-af77-3f78274db7fb"). InnerVolumeSpecName "kube-api-access-khkjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:34.742630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.742606 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 22:00:34.817915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.817883 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\" (UniqueName: \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\") pod \"25bb7c59-7b35-4c04-b920-dda37b605c05\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " Nov 01 22:00:34.818095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.818046 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\") pod \"25bb7c59-7b35-4c04-b920-dda37b605c05\" (UID: \"25bb7c59-7b35-4c04-b920-dda37b605c05\") " Nov 01 22:00:34.818429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.818406 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume" pod "25bb7c59-7b35-4c04-b920-dda37b605c05" (UID: "25bb7c59-7b35-4c04-b920-dda37b605c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:00:34.818764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.818740 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bb7c59-7b35-4c04-b920-dda37b605c05-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:34.818764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.818758 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkjm\" (UniqueName: \"kubernetes.io/projected/c9c74556-6993-48dd-af77-3f78274db7fb-kube-api-access-khkjm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:34.820286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.820258 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw" pod "25bb7c59-7b35-4c04-b920-dda37b605c05" (UID: "25bb7c59-7b35-4c04-b920-dda37b605c05"). InnerVolumeSpecName "kube-api-access-fbkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:00:34.920546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.920479 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkxw\" (UniqueName: \"kubernetes.io/projected/25bb7c59-7b35-4c04-b920-dda37b605c05-kube-api-access-fbkxw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:00:34.983911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.983878 4395 generic.go:356] "Generic (PLEG): container finished" podID="c9c74556-6993-48dd-af77-3f78274db7fb" containerID="d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733" exitCode=0 Nov 01 22:00:34.984076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.983924 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" Nov 01 22:00:34.984076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.983967 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" event={"ID":"c9c74556-6993-48dd-af77-3f78274db7fb","Type":"ContainerDied","Data":"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733"} Nov 01 22:00:34.984076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.984040 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f" event={"ID":"c9c74556-6993-48dd-af77-3f78274db7fb","Type":"ContainerDied","Data":"672e330d037396aa70f23b6700c9ecf905fe550d94dcfd92200b01c1140e04b2"} Nov 01 22:00:34.984233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.984075 4395 scope.go:122] "RemoveContainer" containerID="d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733" Nov 01 22:00:34.985414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.985391 4395 generic.go:356] "Generic (PLEG): container finished" podID="25bb7c59-7b35-4c04-b920-dda37b605c05" containerID="d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417" exitCode=0 Nov 01 22:00:34.985518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.985452 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" Nov 01 22:00:34.985518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.985476 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" event={"ID":"25bb7c59-7b35-4c04-b920-dda37b605c05","Type":"ContainerDied","Data":"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417"} Nov 01 22:00:34.985627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.985521 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2" event={"ID":"25bb7c59-7b35-4c04-b920-dda37b605c05","Type":"ContainerDied","Data":"45d1b1c2a6090dc8ffde734043a035522faa214970a9faf7bbbfd1dd74588ddf"} Nov 01 22:00:34.998815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.998793 4395 scope.go:122] "RemoveContainer" containerID="d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733" Nov 01 22:00:34.999107 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:34.999087 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733\": container with ID starting with d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733 not found: ID does not exist" containerID="d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733" Nov 01 22:00:34.999195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.999114 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733"} err="failed to get container status \"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733\": rpc error: code = NotFound desc = could not find container \"d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733\": container with ID starting with d634f2e8c29375203658ce0d10022b56a19a12eea595798638478219b293e733 not found: ID does not exist" Nov 01 22:00:34.999195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:34.999133 4395 scope.go:122] "RemoveContainer" containerID="d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417" Nov 01 22:00:35.011838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.011820 4395 scope.go:122] "RemoveContainer" containerID="d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417" Nov 01 22:00:35.012181 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:35.012162 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417\": container with ID starting with d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417 not found: ID does not exist" containerID="d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417" Nov 01 22:00:35.012236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.012190 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417"} err="failed to get container status \"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417\": rpc error: code = NotFound desc = could not find container \"d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417\": container with ID starting with d4d775af0e91a77e529d48cea114f3e39e7321263fe1637bb46e5ba30b0a3417 not found: ID does not exist" Nov 01 22:00:35.022155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.022106 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 22:00:35.026563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.026543 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d5713372-57cf5fd967-rnrv2"] Nov 01 22:00:35.034988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.034970 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 22:00:35.040211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.040190 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d5713372-5555895f6-lk42f"] Nov 01 22:00:35.220127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.219949 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bb7c59-7b35-4c04-b920-dda37b605c05" path="/var/lib/kubelet/pods/25bb7c59-7b35-4c04-b920-dda37b605c05/volumes" Nov 01 22:00:35.220700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:35.220680 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c74556-6993-48dd-af77-3f78274db7fb" path="/var/lib/kubelet/pods/c9c74556-6993-48dd-af77-3f78274db7fb/volumes" Nov 01 22:00:38.475220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.475180 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:00:38.476252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476229 4395 cpu_manager.go:414] "containerMap: removing container" podUID="25bb7c59-7b35-4c04-b920-dda37b605c05" containerName="payload-pre-processing" Nov 01 22:00:38.476252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476254 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bb7c59-7b35-4c04-b920-dda37b605c05" containerName="payload-pre-processing" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476275 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476284 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476301 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c9c74556-6993-48dd-af77-3f78274db7fb" containerName="payload-processing" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476308 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c74556-6993-48dd-af77-3f78274db7fb" containerName="payload-processing" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476329 4395 cpu_manager.go:414] "containerMap: removing container" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476335 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94731ae6-1efa-4e04-8848-92bcf6a5f771" containerName="cleanup" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476351 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:38.476465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476359 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:38.476879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476562 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bb7c59-7b35-4c04-b920-dda37b605c05" containerName="payload-pre-processing" Nov 01 22:00:38.476879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476580 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c74556-6993-48dd-af77-3f78274db7fb" containerName="payload-processing" Nov 01 22:00:38.476879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476600 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba0e63c-50d1-492c-86aa-2a5ba04a0e90" containerName="cleanup" Nov 01 22:00:38.476879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.476610 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ed42dd-bf96-4996-a67d-b87fc9d6d38e" containerName="cleanup" Nov 01 22:00:38.482739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.482716 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.501852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.493632 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:00:38.540247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.538491 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-f758d9-dockercfg-4djg7\"" Nov 01 22:00:38.540247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.538855 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-f758d9\"" Nov 01 22:00:38.574057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.553232 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ldp\" (UniqueName: \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.574057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.553577 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.660054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.659347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.660054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.659540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ldp\" (UniqueName: \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.664051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.661276 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.726298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.725594 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ldp\" (UniqueName: \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\") pod \"payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:38.824053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:38.819363 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:39.309424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:39.309395 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:00:39.309999 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:00:39.309973 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe4562_493a_42b1_91b5_9a237a4426bd.slice/crio-444f9866f19ffa51b5359acfd7f764db6e4281a510b2336f6c132cae3b56e70a WatchSource:0}: Error finding container 444f9866f19ffa51b5359acfd7f764db6e4281a510b2336f6c132cae3b56e70a: Status 404 returned error can't find the container with id 444f9866f19ffa51b5359acfd7f764db6e4281a510b2336f6c132cae3b56e70a Nov 01 22:00:40.072953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:40.072910 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" event={"ID":"fefe4562-493a-42b1-91b5-9a237a4426bd","Type":"ContainerStarted","Data":"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545"} Nov 01 22:00:40.073536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:40.072965 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" event={"ID":"fefe4562-493a-42b1-91b5-9a237a4426bd","Type":"ContainerStarted","Data":"444f9866f19ffa51b5359acfd7f764db6e4281a510b2336f6c132cae3b56e70a"} Nov 01 22:00:40.073536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:40.073091 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:40.093188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:40.093146 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" podStartSLOduration=2.093130358 podStartE2EDuration="2.093130358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:40.09180158 +0000 UTC m=+987.065704118" watchObservedRunningTime="2026-11-01 22:00:40.093130358 +0000 UTC m=+987.067032899" Nov 01 22:00:42.945421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:42.945393 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:00:49.592655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:49.592614 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 01 22:00:49.740338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:49.740295 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:00:49.792090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:49.783118 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:00:51.083126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:51.083096 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:00:52.302679 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:00:52.302637 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:00:53.025214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:53.025185 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf"] Nov 01 22:00:53.030396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:53.030371 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa4642-6wzbf"] Nov 01 22:00:53.167514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:53.167468 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31184b82-af95-480e-b003-033b5bd67260" path="/var/lib/kubelet/pods/31184b82-af95-480e-b003-033b5bd67260/volumes" Nov 01 22:00:58.080774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.080734 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:00:58.088481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.088456 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.107435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.107411 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.208006 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2c5135-openshift-default-dockercfg-dfww7\"" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209353 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209437 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqc2n\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209470 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209580 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209706 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209776 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209824 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209891 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.224059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.209934 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324398 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324548 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324619 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324667 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324730 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324771 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324848 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324920 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqc2n\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.324947 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.332264 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.343330 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.343386 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.343794 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.354437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.354369 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.409065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.407431 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.420064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.413636 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqc2n\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.447057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.420715 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.447317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.430952 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\") pod \"e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:58.716177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:58.711352 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:00:59.038246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.038210 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:00:59.038373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.038284 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:00:59.038373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.038322 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:00:59.040178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.040154 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:00:59.811570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.811497 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerStarted","Data":"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38"} Nov 01 22:00:59.813840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.813796 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerStarted","Data":"fe55b44b08e9fb46a39d00c2d19893da82652165fdcad44f81cdf8a642e3d125"} Nov 01 22:00:59.887795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:00:59.887726 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podStartSLOduration=1.887706663 podStartE2EDuration="1.887706663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:59.83949268 +0000 UTC m=+1006.813395223" watchObservedRunningTime="2026-11-01 22:00:59.887706663 +0000 UTC m=+1006.861609202" Nov 01 22:01:00.343708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.343676 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:01:00.348089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.348069 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.353833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.353807 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-e2bf63-dockercfg-w9c8k\"" Nov 01 22:01:00.353969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.353830 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-e2bf63\"" Nov 01 22:01:00.359887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.359862 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:01:00.409961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.409934 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wmv\" (UniqueName: \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.410290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.410261 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.537055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.514154 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.589665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.586674 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wmv\" (UniqueName: \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.589665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.538170 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.712434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.711054 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wmv\" (UniqueName: \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\") pod \"payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:00.750048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.739104 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:01:00.759098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.759061 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:01:00.869255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.869227 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:01:00.870628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.870603 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:01:00.965901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:00.965825 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:01.519201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:01.500138 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:01:01.876735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:01.876644 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" event={"ID":"7f7804bb-aec3-4506-a386-008177911e3f","Type":"ContainerStarted","Data":"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77"} Nov 01 22:01:01.876735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:01.876701 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" event={"ID":"7f7804bb-aec3-4506-a386-008177911e3f","Type":"ContainerStarted","Data":"949a5b47d695df0ea61c643e9d1d8225f589117507ddda9b6d3caf393394efde"} Nov 01 22:01:01.877231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:01.876749 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:01.896890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:01.896832 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" podStartSLOduration=1.896814411 podStartE2EDuration="1.896814411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:01.89583485 +0000 UTC m=+1008.869737393" watchObservedRunningTime="2026-11-01 22:01:01.896814411 +0000 UTC m=+1008.870716951" Nov 01 22:01:06.066054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:06.064002 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt"] Nov 01 22:01:06.073053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:06.069021 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76907f68-s2mtt"] Nov 01 22:01:07.165763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:07.165733 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19452e5d-eda5-40ce-a174-9d0f94db5c46" path="/var/lib/kubelet/pods/19452e5d-eda5-40ce-a174-9d0f94db5c46/volumes" Nov 01 22:01:10.666149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.666115 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:01:10.673993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.673757 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.679563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.679542 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-2c5135-serving-cert\"" Nov 01 22:01:10.684063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.684020 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:01:10.794594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.794563 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.794761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.794743 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.794821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.794789 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6cv\" (UniqueName: \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.794939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.794916 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.897475 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.897553 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6cv\" (UniqueName: \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.897635 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.897783 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.898639 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.905212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.902833 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.911663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.911629 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.925410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.925120 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6cv\" (UniqueName: \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\") pod \"maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:10.997475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:10.997425 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:11.391125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:11.384597 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:01:11.463067 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:01:11.462978 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a487d68_697d_4d12_bd59_1ce78c41e084.slice/crio-2bffe4251eaa01d20f7d942a2e9c5d7f53c63b687fc80e83b22cd45cd3568f47 WatchSource:0}: Error finding container 2bffe4251eaa01d20f7d942a2e9c5d7f53c63b687fc80e83b22cd45cd3568f47: Status 404 returned error can't find the container with id 2bffe4251eaa01d20f7d942a2e9c5d7f53c63b687fc80e83b22cd45cd3568f47 Nov 01 22:01:11.476304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:11.474477 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:01:12.123289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:12.123256 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" event={"ID":"1a487d68-697d-4d12-bd59-1ce78c41e084","Type":"ContainerStarted","Data":"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b"} Nov 01 22:01:12.123655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:12.123314 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" event={"ID":"1a487d68-697d-4d12-bd59-1ce78c41e084","Type":"ContainerStarted","Data":"2bffe4251eaa01d20f7d942a2e9c5d7f53c63b687fc80e83b22cd45cd3568f47"} Nov 01 22:01:12.123655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:12.123442 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:12.145435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:12.145393 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" podStartSLOduration=1.7119864310000001 podStartE2EDuration="2.145380606s" totalImagesPullingTime="433.394175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:12.142513932 +0000 UTC m=+1019.116416473" watchObservedRunningTime="2026-11-01 22:01:12.145380606 +0000 UTC m=+1019.119283171" Nov 01 22:01:12.887781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:12.887758 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:01:18.157093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:18.156214 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:01:22.314972 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:01:22.314095 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:01:29.892493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:29.892417 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:01:29.896959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:29.896936 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:29.905940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:29.905911 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-21dd16de\"" Nov 01 22:01:29.912525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:29.912489 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:01:30.004416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.004383 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntw2d\" (UniqueName: \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.004571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.004552 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.108053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.107431 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntw2d\" (UniqueName: \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.108053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.107620 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.118050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.108643 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.148116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.146411 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntw2d\" (UniqueName: \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.228057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.227308 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:30.731975 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:01:30.731942 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3399e42_4ac0_4d38_86cf_355722423ea4.slice/crio-19a2e8d31383724983d6d7bcc2fc511a6f7d717e571620248ada5e33c98387a8 WatchSource:0}: Error finding container 19a2e8d31383724983d6d7bcc2fc511a6f7d717e571620248ada5e33c98387a8: Status 404 returned error can't find the container with id 19a2e8d31383724983d6d7bcc2fc511a6f7d717e571620248ada5e33c98387a8 Nov 01 22:01:30.732196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:30.732174 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:01:31.666622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:31.666586 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" event={"ID":"a3399e42-4ac0-4d38-86cf-355722423ea4","Type":"ContainerStarted","Data":"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed"} Nov 01 22:01:31.667253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:31.667225 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" event={"ID":"a3399e42-4ac0-4d38-86cf-355722423ea4","Type":"ContainerStarted","Data":"19a2e8d31383724983d6d7bcc2fc511a6f7d717e571620248ada5e33c98387a8"} Nov 01 22:01:31.668091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:31.668066 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:31.695447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:31.695401 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" podStartSLOduration=2.695389637 podStartE2EDuration="2.695389637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:31.690698701 +0000 UTC m=+1038.664601240" watchObservedRunningTime="2026-11-01 22:01:31.695389637 +0000 UTC m=+1038.669292176" Nov 01 22:01:42.677007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:42.676981 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:01:52.533056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:52.523276 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.566266 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:01:52.560242 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:01:52.750054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:52.749872 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:52.884104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:52.883099 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:01:52.888155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:52.887681 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:01:53.318073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:53.277156 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:53.318073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:53.292661 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:53.335606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:53.335149 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.171785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:54.163626 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.287288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:54.281813 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:54.311979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:54.301556 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:01:54.670569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:01:54.670532 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:02:00.921115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.906229 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:00.921115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.906312 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:00.936958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.930694 4395 generic.go:356] "Generic (PLEG): container finished" podID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerID="87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38" exitCode=0 Nov 01 22:02:00.936958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.930814 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerDied","Data":"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38"} Nov 01 22:02:00.936958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.931508 4395 scope.go:122] "RemoveContainer" containerID="87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38" Nov 01 22:02:00.962053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.952537 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:02:00.962053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.952638 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:02:00.962053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:00.952690 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:02:01.940764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:01.940727 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerStarted","Data":"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9"} Nov 01 22:02:02.713297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:02.713255 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:02:02.713979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:02.713954 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:02.714278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:02.714237 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:03.713247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:03.713202 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:03.713680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:03.713262 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:04.712747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:04.712716 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:04.712900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:04.712778 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:05.712854 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:05.712822 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:05.713276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:05.712875 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:06.712463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:06.712422 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 01 22:02:06.712655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:06.712485 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 01 22:02:07.717459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:07.717425 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:02:07.717901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:07.717884 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:02:07.718727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:07.718707 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:02:14.501464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:14.501432 4395 scope.go:122] "RemoveContainer" containerID="828223c8bd4fca16b5b6d531bd0d6be7eb28158fa104b2fe5cbc618ac04bf9b3" Nov 01 22:02:14.511713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:14.511692 4395 scope.go:122] "RemoveContainer" containerID="9a42fdcbcb88d05bcc918feacc39bce3750eb595001caab6ab541be1df0a9b28" Nov 01 22:02:22.397150 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:02:22.397110 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:02:45.369969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.369923 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:02:45.375892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.375870 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.391219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.391193 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b1b5598d-dockercfg-8gb7n\"" Nov 01 22:02:45.391719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.391480 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:02:45.551051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.543827 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b1b5598d\"" Nov 01 22:02:45.577470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.577428 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zln5w\" (UniqueName: \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.577989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.577963 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.704347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.694451 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.704347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.694789 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zln5w\" (UniqueName: \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.704347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.696768 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:45.763051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:45.733641 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zln5w\" (UniqueName: \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\") pod \"payload-processing-e2e-mt-b1b5598d-684b864db8-46b66\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:46.001000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.000962 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:46.326332 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.326304 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:02:46.890057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.889589 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" event={"ID":"60920858-f25c-4d5c-9e5d-dda600494577","Type":"ContainerStarted","Data":"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843"} Nov 01 22:02:46.890057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.889660 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" event={"ID":"60920858-f25c-4d5c-9e5d-dda600494577","Type":"ContainerStarted","Data":"2a6b81ce74dd0f7cedeab2a66e878e10604200f87675fcee57eabe8e29f734ba"} Nov 01 22:02:46.898937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.890946 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:02:46.910688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:46.910627 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" podStartSLOduration=1.910613747 podStartE2EDuration="1.910613747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:02:46.90896589 +0000 UTC m=+1113.882868424" watchObservedRunningTime="2026-11-01 22:02:46.910613747 +0000 UTC m=+1113.884516286" Nov 01 22:02:52.330913 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:02:52.330872 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:02:58.906927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:02:58.906847 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:03:03.376553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:03.376511 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 22:03:03.390498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:03.390455 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" containerID="cri-o://ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f" gracePeriod=30 Nov 01 22:03:08.705982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.705578 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 22:03:08.825180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825150 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825193 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825248 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825291 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825346 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825446 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825479 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825478 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:08.825587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825534 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825535 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:08.825846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825688 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\") pod \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\" (UID: \"23d0891c-109e-46b5-ae68-fec2aa6d4b44\") " Nov 01 22:03:08.825846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825736 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:08.825846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.825813 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:08.826069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826008 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:08.826655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826632 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.826759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826660 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.826759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826677 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.826759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826691 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.826759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.826703 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.827814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.827787 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:03:08.827964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.827943 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:08.828264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.828246 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "kube-api-access-tmrw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:08.828338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.828293 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token" pod "23d0891c-109e-46b5-ae68-fec2aa6d4b44" (UID: "23d0891c-109e-46b5-ae68-fec2aa6d4b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:08.938164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.931124 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.938164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.931159 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.938164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.931175 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrw8\" (UniqueName: \"kubernetes.io/projected/23d0891c-109e-46b5-ae68-fec2aa6d4b44-kube-api-access-tmrw8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:08.938164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:08.931191 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0891c-109e-46b5-ae68-fec2aa6d4b44-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:09.172355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.172308 4395 generic.go:356] "Generic (PLEG): container finished" podID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerID="ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f" exitCode=0 Nov 01 22:03:09.172551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.172373 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" Nov 01 22:03:09.172551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.172395 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" event={"ID":"23d0891c-109e-46b5-ae68-fec2aa6d4b44","Type":"ContainerDied","Data":"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f"} Nov 01 22:03:09.172551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.172451 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7" event={"ID":"23d0891c-109e-46b5-ae68-fec2aa6d4b44","Type":"ContainerDied","Data":"1dd4384912cbede290a1eee2885ec3f18e5fd0501aae4465206bed1c7503500d"} Nov 01 22:03:09.172551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.172477 4395 scope.go:122] "RemoveContainer" containerID="ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f" Nov 01 22:03:09.187141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.187122 4395 scope.go:122] "RemoveContainer" containerID="ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f" Nov 01 22:03:09.187465 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:03:09.187441 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f\": container with ID starting with ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f not found: ID does not exist" containerID="ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f" Nov 01 22:03:09.187567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.187470 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f"} err="failed to get container status \"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f\": rpc error: code = NotFound desc = could not find container \"ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f\": container with ID starting with ee0e931e75aef62cc3a30959accf9d2120f6865092f3d991db360f1f13fe2f7f not found: ID does not exist" Nov 01 22:03:09.197882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.197859 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 22:03:09.204056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:09.204016 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-sbgs7"] Nov 01 22:03:11.169441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:11.169406 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" path="/var/lib/kubelet/pods/23d0891c-109e-46b5-ae68-fec2aa6d4b44/volumes" Nov 01 22:03:22.303950 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:03:22.303914 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:03:33.509890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:33.509858 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 22:03:33.510337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:33.510066 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" containerID="cri-o://363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73" gracePeriod=30 Nov 01 22:03:36.032051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:36.032002 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv"] Nov 01 22:03:36.035757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:36.035735 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ec4b36-2gwvv"] Nov 01 22:03:37.270079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:37.270042 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24ea02b-5ba7-4868-b2d3-e7fa36c21842" path="/var/lib/kubelet/pods/f24ea02b-5ba7-4868-b2d3-e7fa36c21842/volumes" Nov 01 22:03:38.951120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:38.951096 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 22:03:39.039517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039484 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.039704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039602 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.039704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039638 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.039704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039678 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.039890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039802 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.039890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039874 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.040000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039911 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:03:39.040000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.039941 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.040137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.040013 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.040137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.040107 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\") pod \"66a04d67-3da0-4336-b86a-b13d2701b947\" (UID: \"66a04d67-3da0-4336-b86a-b13d2701b947\") " Nov 01 22:03:39.040866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.040607 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:39.040866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.040819 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:39.041260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.041235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:39.041353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.041246 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.041353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.041286 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a04d67-3da0-4336-b86a-b13d2701b947-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.041353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.041303 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.041524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.041501 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:39.042556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.042520 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "kube-api-access-fbd4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:39.043078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.043051 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:03:39.043764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.043739 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:03:39.044547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.044531 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token" pod "66a04d67-3da0-4336-b86a-b13d2701b947" (UID: "66a04d67-3da0-4336-b86a-b13d2701b947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:03:39.142994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.142968 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.142994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.142992 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a04d67-3da0-4336-b86a-b13d2701b947-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.143175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.143001 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.143175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.143011 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.143175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.143019 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbd4p\" (UniqueName: \"kubernetes.io/projected/66a04d67-3da0-4336-b86a-b13d2701b947-kube-api-access-fbd4p\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.143175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.143043 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a04d67-3da0-4336-b86a-b13d2701b947-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:03:39.545908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.545870 4395 generic.go:356] "Generic (PLEG): container finished" podID="66a04d67-3da0-4336-b86a-b13d2701b947" containerID="363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73" exitCode=0 Nov 01 22:03:39.546102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.545918 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" Nov 01 22:03:39.546102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.545953 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" event={"ID":"66a04d67-3da0-4336-b86a-b13d2701b947","Type":"ContainerDied","Data":"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73"} Nov 01 22:03:39.546102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.546013 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876" event={"ID":"66a04d67-3da0-4336-b86a-b13d2701b947","Type":"ContainerDied","Data":"d9402646f129bbbd5847fe304552a1b04a474764cef2b3ff0c67865c417fe277"} Nov 01 22:03:39.546102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.546054 4395 scope.go:122] "RemoveContainer" containerID="363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73" Nov 01 22:03:39.556597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.556575 4395 scope.go:122] "RemoveContainer" containerID="363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73" Nov 01 22:03:39.556903 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:03:39.556879 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73\": container with ID starting with 363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73 not found: ID does not exist" containerID="363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73" Nov 01 22:03:39.556962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.556912 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73"} err="failed to get container status \"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73\": rpc error: code = NotFound desc = could not find container \"363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73\": container with ID starting with 363f4e401959de61b0958476ed95bb9023c5956afc22430f10e6d7e116fefe73 not found: ID does not exist" Nov 01 22:03:39.568211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.568188 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 22:03:39.572863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:39.572841 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-ks876"] Nov 01 22:03:41.247910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:41.247862 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" path="/var/lib/kubelet/pods/66a04d67-3da0-4336-b86a-b13d2701b947/volumes" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.583982 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596426 4395 cpu_manager.go:414] "containerMap: removing container" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596454 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596472 4395 cpu_manager.go:414] "containerMap: removing container" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596480 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596686 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a04d67-3da0-4336-b86a-b13d2701b947" containerName="istio-proxy" Nov 01 22:03:42.609056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.596701 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0891c-109e-46b5-ae68-fec2aa6d4b44" containerName="istio-proxy" Nov 01 22:03:42.797590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.797562 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:42.827492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.825428 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d506c6\"" Nov 01 22:03:42.827492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.825797 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d506c6-dockercfg-8k85h\"" Nov 01 22:03:42.835173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.835136 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:42.856147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.856067 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:03:42.955594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.944564 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:42.973111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:42.970195 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtb4\" (UniqueName: \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.077115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.077081 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:03:43.122102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.108859 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.122102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.108950 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtb4\" (UniqueName: \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.123996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.123971 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.138050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.130890 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.215990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.200988 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtb4\" (UniqueName: \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\") pod \"payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.336054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.329107 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:43.376188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.363352 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:43.421706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.421660 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.430404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.430377 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h82s\" (UniqueName: \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.450621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.441070 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:03:43.558057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.539145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h82s\" (UniqueName: \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.558563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.558531 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.598053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.567007 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.738709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.738583 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h82s\" (UniqueName: \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\") pod \"payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:43.758625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.757649 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:43.758625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.758201 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:03:43.834447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.833146 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:43.848218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.837414 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:43.848218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.840199 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:03:43.929307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:43.929272 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:44.540259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.494548 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:44.588461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.588381 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:03:44.815457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.807422 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:44.815457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.807834 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:03:44.815457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.808162 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:03:44.815457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.808215 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:03:44.852991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.852953 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:03:44.864055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.853332 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:03:44.893291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.884941 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:44.893291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.889543 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:03:44.968408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.966158 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:03:44.985072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:44.969321 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:03:45.074267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.073950 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:03:45.127297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.126637 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:03:45.273753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.271658 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:03:45.273753 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:03:45.272448 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596e5fb4_9f8d_44a7_926e_bf8995c60e52.slice/crio-b12e838b2560f38f7d17089d96f6504f4eaa2b682e8843c32745197cba4a0b06 WatchSource:0}: Error finding container b12e838b2560f38f7d17089d96f6504f4eaa2b682e8843c32745197cba4a0b06: Status 404 returned error can't find the container with id b12e838b2560f38f7d17089d96f6504f4eaa2b682e8843c32745197cba4a0b06 Nov 01 22:03:45.909871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.909838 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" event={"ID":"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0","Type":"ContainerStarted","Data":"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9"} Nov 01 22:03:45.910375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.909895 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" event={"ID":"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0","Type":"ContainerStarted","Data":"41c0b86e03c5e7efbb4350d24ee4c31968fe0751b996ba926007c7ca56645ad9"} Nov 01 22:03:45.910375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.910055 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:45.911693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.911668 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" event={"ID":"596e5fb4-9f8d-44a7-926e-bf8995c60e52","Type":"ContainerStarted","Data":"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31"} Nov 01 22:03:45.911814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.911710 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" event={"ID":"596e5fb4-9f8d-44a7-926e-bf8995c60e52","Type":"ContainerStarted","Data":"b12e838b2560f38f7d17089d96f6504f4eaa2b682e8843c32745197cba4a0b06"} Nov 01 22:03:45.911814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.911752 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:45.933939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.933897 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" podStartSLOduration=3.933885914 podStartE2EDuration="3.933885914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:45.932204936 +0000 UTC m=+1172.906107475" watchObservedRunningTime="2026-11-01 22:03:45.933885914 +0000 UTC m=+1172.907788453" Nov 01 22:03:45.955702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:45.955660 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" podStartSLOduration=3.955647555 podStartE2EDuration="3.955647555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:45.953099038 +0000 UTC m=+1172.927001577" watchObservedRunningTime="2026-11-01 22:03:45.955647555 +0000 UTC m=+1172.929550093" Nov 01 22:03:46.231976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.230469 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:03:46.239924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.239857 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.254556 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.255734 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.255794 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.255826 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4l5x\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.255942 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.255976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.256004 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.256084 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.256211 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a19582-openshift-default-dockercfg-k8g64\"" Nov 01 22:03:46.256268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.256233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.256793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.256312 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.367660 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.367813 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.367864 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.367890 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4l5x\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.367997 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.368043 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.368068 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.368146 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.368379 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.368865 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.369874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.369400 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.370126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.370092 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.373213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.370895 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.373213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.371247 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.435057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.418722 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.436567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.435887 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.436742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.436486 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4l5x\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.479383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.479346 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\") pod \"e2e-api-a-a19582-openshift-default-7df57db669-fzn8r\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.573052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.572367 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:46.975204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.975179 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:03:46.977297 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:03:46.977256 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea0c9e9_ce17_413a_9efa_feaaf0f57802.slice/crio-a7d63cf893f73a238fb25235b688d69a40940f45e1f8fb5a5289bacd3dd1c1aa WatchSource:0}: Error finding container a7d63cf893f73a238fb25235b688d69a40940f45e1f8fb5a5289bacd3dd1c1aa: Status 404 returned error can't find the container with id a7d63cf893f73a238fb25235b688d69a40940f45e1f8fb5a5289bacd3dd1c1aa Nov 01 22:03:46.980473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.980436 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:03:46.980569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.980517 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:03:46.980569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:46.980558 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:03:47.938099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:47.938059 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerStarted","Data":"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c"} Nov 01 22:03:47.938283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:47.938163 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerStarted","Data":"a7d63cf893f73a238fb25235b688d69a40940f45e1f8fb5a5289bacd3dd1c1aa"} Nov 01 22:03:47.994053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:47.993478 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podStartSLOduration=1.9934602780000001 podStartE2EDuration="1.993460278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:47.992268164 +0000 UTC m=+1174.966170741" watchObservedRunningTime="2026-11-01 22:03:47.993460278 +0000 UTC m=+1174.967362820" Nov 01 22:03:48.573181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:48.573153 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:48.578911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:48.578885 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:48.947083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:48.946960 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:48.948066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:48.948006 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:03:50.051613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:50.051577 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:03:52.303065 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:03:52.302987 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:03:54.834337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:54.834302 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:03:54.841545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:54.841522 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:54.862643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:54.862614 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d506c6-serving-cert\"" Nov 01 22:03:54.868621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:54.868590 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:03:55.000642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.000611 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.000788 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.000653 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdwg\" (UniqueName: \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.000832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.000793 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.000916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.000902 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.061056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.060171 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" containerID="cri-o://43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed" gracePeriod=25 Nov 01 22:03:55.111082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.104141 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.111082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.104387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.111082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.104437 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdwg\" (UniqueName: \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.111082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.104569 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.111082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.105521 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.155050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.154602 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.164053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.156301 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdwg\" (UniqueName: \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.164053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.156843 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\") pod \"maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.180053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.177614 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:55.655087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:55.652435 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:03:55.667906 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:03:55.667872 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed79f4b_8fbd_4018_8516_80e127799b53.slice/crio-489b943c75573aa46018b6dbbd1c413080c9c83841f80303b372a10a45f9e3d7 WatchSource:0}: Error finding container 489b943c75573aa46018b6dbbd1c413080c9c83841f80303b372a10a45f9e3d7: Status 404 returned error can't find the container with id 489b943c75573aa46018b6dbbd1c413080c9c83841f80303b372a10a45f9e3d7 Nov 01 22:03:56.124214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:56.124179 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" event={"ID":"3ed79f4b-8fbd-4018-8516-80e127799b53","Type":"ContainerStarted","Data":"489b943c75573aa46018b6dbbd1c413080c9c83841f80303b372a10a45f9e3d7"} Nov 01 22:03:56.921798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:56.921768 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:03:56.922099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:56.922080 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:03:57.134010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:57.133974 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" event={"ID":"3ed79f4b-8fbd-4018-8516-80e127799b53","Type":"ContainerStarted","Data":"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5"} Nov 01 22:03:57.134483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:57.134088 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:03:57.158252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:03:57.158200 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" podStartSLOduration=2.682707815 podStartE2EDuration="3.158185081s" totalImagesPullingTime="475.477266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:57.154538648 +0000 UTC m=+1184.128441184" watchObservedRunningTime="2026-11-01 22:03:57.158185081 +0000 UTC m=+1184.132087622" Nov 01 22:04:02.675421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:02.675382 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:04:03.277320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:03.277291 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:04:07.876113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:07.864149 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:04:08.317788 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.270558 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:04:08.355215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.270710 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:04:08.447610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.355792 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:04:08.464023 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.359732 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:08.627401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.616328 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:04:08.661338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.648586 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-edd32a-dockercfg-kxn95\"" Nov 01 22:04:08.722384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.721807 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:04:08.732592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.732561 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:08.755282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.755252 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edd32a\"" Nov 01 22:04:08.788252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.788218 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 22:04:08.888266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.888189 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:04:08.922713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.922683 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:08.922996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:08.922976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fnl\" (UniqueName: \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.035774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.035738 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.036092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.036067 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fnl\" (UniqueName: \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.072342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.039760 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.098645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.097141 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fnl\" (UniqueName: \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\") pod \"payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.230166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.227365 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.594183 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:09.594152 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca03fd1e_f8d3_4c7a_bc29_e569a356c7fe.slice/crio-6634d575abbd8dd9519149cf2b322a9f5aa2b829ba876f6c6ca31732a6da0882 WatchSource:0}: Error finding container 6634d575abbd8dd9519149cf2b322a9f5aa2b829ba876f6c6ca31732a6da0882: Status 404 returned error can't find the container with id 6634d575abbd8dd9519149cf2b322a9f5aa2b829ba876f6c6ca31732a6da0882 Nov 01 22:04:09.594859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.594829 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:04:09.943170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.942185 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" event={"ID":"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe","Type":"ContainerStarted","Data":"e4a11825d8a17c3e72132c95157a0d464c14d660ffbb7bfcec2fc0955b3d2d79"} Nov 01 22:04:09.943170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.942274 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" event={"ID":"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe","Type":"ContainerStarted","Data":"6634d575abbd8dd9519149cf2b322a9f5aa2b829ba876f6c6ca31732a6da0882"} Nov 01 22:04:09.943696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.943509 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:09.965621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:09.965576 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" podStartSLOduration=2.96556294 podStartE2EDuration="2.96556294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:09.96135901 +0000 UTC m=+1196.935261549" watchObservedRunningTime="2026-11-01 22:04:09.96556294 +0000 UTC m=+1196.939465509" Nov 01 22:04:12.679053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:12.678449 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:04:15.539068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.539022 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:04:15.704155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.704122 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\") pod \"a3399e42-4ac0-4d38-86cf-355722423ea4\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " Nov 01 22:04:15.704342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.704164 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\" (UniqueName: \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\") pod \"a3399e42-4ac0-4d38-86cf-355722423ea4\" (UID: \"a3399e42-4ac0-4d38-86cf-355722423ea4\") " Nov 01 22:04:15.704582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.704558 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume" pod "a3399e42-4ac0-4d38-86cf-355722423ea4" (UID: "a3399e42-4ac0-4d38-86cf-355722423ea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:15.704854 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.704837 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3399e42-4ac0-4d38-86cf-355722423ea4-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:15.706448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.706423 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d" pod "a3399e42-4ac0-4d38-86cf-355722423ea4" (UID: "a3399e42-4ac0-4d38-86cf-355722423ea4"). InnerVolumeSpecName "kube-api-access-ntw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:15.806266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:15.806239 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntw2d\" (UniqueName: \"kubernetes.io/projected/a3399e42-4ac0-4d38-86cf-355722423ea4-kube-api-access-ntw2d\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:16.113733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.113648 4395 generic.go:356] "Generic (PLEG): container finished" podID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerID="43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed" exitCode=0 Nov 01 22:04:16.113733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.113698 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" Nov 01 22:04:16.113925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.113730 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" event={"ID":"a3399e42-4ac0-4d38-86cf-355722423ea4","Type":"ContainerDied","Data":"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed"} Nov 01 22:04:16.113925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.113793 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885" event={"ID":"a3399e42-4ac0-4d38-86cf-355722423ea4","Type":"ContainerDied","Data":"19a2e8d31383724983d6d7bcc2fc511a6f7d717e571620248ada5e33c98387a8"} Nov 01 22:04:16.113925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.113824 4395 scope.go:122] "RemoveContainer" containerID="43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed" Nov 01 22:04:16.125561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.125533 4395 scope.go:122] "RemoveContainer" containerID="43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed" Nov 01 22:04:16.125836 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:16.125816 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed\": container with ID starting with 43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed not found: ID does not exist" containerID="43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed" Nov 01 22:04:16.125884 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.125844 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed"} err="failed to get container status \"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed\": rpc error: code = NotFound desc = could not find container \"43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed\": container with ID starting with 43375778f4added4377fd500302e546a617928c652027adc9de1f2ab2f4408ed not found: ID does not exist" Nov 01 22:04:16.140900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.140875 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:04:16.144976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:16.144954 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-4j885"] Nov 01 22:04:17.166236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:17.166206 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" path="/var/lib/kubelet/pods/a3399e42-4ac0-4d38-86cf-355722423ea4/volumes" Nov 01 22:04:18.677465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.676414 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 22:04:18.677465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.676711 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" containerID="cri-o://dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4" gracePeriod=30 Nov 01 22:04:18.689106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.689069 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 22:04:18.689637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.689568 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" containerID="cri-o://b960aeb53472fb592840b1859c83604a39306e207d051a85bd5388763e290380" gracePeriod=30 Nov 01 22:04:18.718197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.717179 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 22:04:18.718197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.717387 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" containerID="cri-o://4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e" gracePeriod=30 Nov 01 22:04:18.722292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.722272 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 22:04:18.722632 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:18.722610 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" containerID="cri-o://2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671" gracePeriod=30 Nov 01 22:04:20.110259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:20.110225 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:04:20.110632 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:20.110298 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:04:20.503102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:20.499223 4395 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:04:20.503102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:20.499294 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:04:22.067049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.066371 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:04:22.333336 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:22.333263 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:04:22.365475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.365440 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:04:22.366603 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.366576 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" Nov 01 22:04:22.366603 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.366600 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" Nov 01 22:04:22.366915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.366838 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3399e42-4ac0-4d38-86cf-355722423ea4" containerName="payload-processing" Nov 01 22:04:22.375310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.375288 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.378642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.378616 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:04:22.379696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.379675 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-edd32a-serving-cert\"" Nov 01 22:04:22.521051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.516477 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.521051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.516644 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wfw\" (UniqueName: \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.521051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.516695 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.521051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.516864 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.623161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.622532 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.623161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.622781 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.623161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.622901 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wfw\" (UniqueName: \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.623161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.622929 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.641650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.624255 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.646917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.645542 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.649904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.647615 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.658008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.657985 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wfw\" (UniqueName: \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\") pod \"maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:22.713997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:22.713976 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:23.121737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:23.121689 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:04:23.332659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:23.332619 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" event={"ID":"673906a5-1463-4b6e-bdf8-6579af0f3175","Type":"ContainerStarted","Data":"97d73b1ec76545f90d4682831a1fc240676ac11d23455f87aaa1539a375bc33b"} Nov 01 22:04:24.015478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.015450 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 22:04:24.057078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.055771 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 22:04:24.128770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.128730 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 22:04:24.161188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161113 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.161406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161386 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.161545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161532 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.161659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161646 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.161773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161759 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.161911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161898 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.162009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.161996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.162207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162189 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.162431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162415 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.162573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162558 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.162672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162660 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.162853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.163001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.162986 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.163127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163113 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.163325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163296 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.163425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163375 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.163469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163445 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\") pod \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\" (UID: \"359f99ba-ea97-4f56-9553-9e1ea84c10ee\") " Nov 01 22:04:24.163504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163480 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\") pod \"6529c812-e3d7-46b9-9320-05958ff0580b\" (UID: \"6529c812-e3d7-46b9-9320-05958ff0580b\") " Nov 01 22:04:24.163566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.163540 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.164385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.164360 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.164773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.164735 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:24.165041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.165007 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.165731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.165703 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.165901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.165882 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.166509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.166481 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "kube-api-access-bdfnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.166752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.166728 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.167121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.167063 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:24.167351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.167328 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.168092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168019 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.168271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168253 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.168574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168490 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "kube-api-access-gnxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.168652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168592 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.168652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168595 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.168652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.168608 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:04:24.170221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.170167 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.170328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.170290 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo" pod "6529c812-e3d7-46b9-9320-05958ff0580b" (UID: "6529c812-e3d7-46b9-9320-05958ff0580b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:04:24.170542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.170520 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy" pod "359f99ba-ea97-4f56-9553-9e1ea84c10ee" (UID: "359f99ba-ea97-4f56-9553-9e1ea84c10ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.269909 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.269992 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270043 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270071 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270114 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270358 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270406 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270471 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.270559 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\") pod \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\" (UID: \"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293\") " Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271328 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271354 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxdr\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-kube-api-access-gnxdr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271372 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271386 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271400 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271415 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271429 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271443 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271457 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271470 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271482 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6529c812-e3d7-46b9-9320-05958ff0580b-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271494 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdfnt\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-kube-api-access-bdfnt\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271509 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6529c812-e3d7-46b9-9320-05958ff0580b-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271522 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6529c812-e3d7-46b9-9320-05958ff0580b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271537 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359f99ba-ea97-4f56-9553-9e1ea84c10ee-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271550 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.276237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.271562 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6529c812-e3d7-46b9-9320-05958ff0580b-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.279152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.278221 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.285529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.283620 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.285529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.284080 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:24.285529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.285254 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.285529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.285471 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.291083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.290988 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.303119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.302279 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.303119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.302345 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "kube-api-access-s2xpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.303451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.303413 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo" pod "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" (UID: "abaa57b6-8fa5-4290-b9b0-e7f3b9b60293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:04:24.345227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.344931 4395 generic.go:356] "Generic (PLEG): container finished" podID="6529c812-e3d7-46b9-9320-05958ff0580b" containerID="2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671" exitCode=0 Nov 01 22:04:24.345227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.345001 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" event={"ID":"6529c812-e3d7-46b9-9320-05958ff0580b","Type":"ContainerDied","Data":"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671"} Nov 01 22:04:24.345227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.345082 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" event={"ID":"6529c812-e3d7-46b9-9320-05958ff0580b","Type":"ContainerDied","Data":"e0569730de380c16ff2bb019eb5d22503b50ad2e9ec35715f08925317c88fec6"} Nov 01 22:04:24.345227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.345116 4395 scope.go:122] "RemoveContainer" containerID="2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671" Nov 01 22:04:24.345227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.345175 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24" Nov 01 22:04:24.356210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.354398 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" event={"ID":"673906a5-1463-4b6e-bdf8-6579af0f3175","Type":"ContainerStarted","Data":"a6e5ec69ff613e4046194df9aaa441ff18fc1f86fa4bdd277331bfc9098f9073"} Nov 01 22:04:24.356210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.354478 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:24.356695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.356674 4395 generic.go:356] "Generic (PLEG): container finished" podID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerID="b960aeb53472fb592840b1859c83604a39306e207d051a85bd5388763e290380" exitCode=0 Nov 01 22:04:24.356839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.356820 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" event={"ID":"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0","Type":"ContainerDied","Data":"b960aeb53472fb592840b1859c83604a39306e207d051a85bd5388763e290380"} Nov 01 22:04:24.358248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.358227 4395 generic.go:356] "Generic (PLEG): container finished" podID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerID="4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e" exitCode=0 Nov 01 22:04:24.359090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.358302 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" event={"ID":"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293","Type":"ContainerDied","Data":"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e"} Nov 01 22:04:24.359090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.358480 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" event={"ID":"abaa57b6-8fa5-4290-b9b0-e7f3b9b60293","Type":"ContainerDied","Data":"5e8505473afe6af7c987f31a855aa7ca6eb75aa775c7945f904d499a020643cc"} Nov 01 22:04:24.359090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.358326 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb" Nov 01 22:04:24.362997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.360646 4395 generic.go:356] "Generic (PLEG): container finished" podID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerID="dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4" exitCode=0 Nov 01 22:04:24.362997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.360744 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" event={"ID":"359f99ba-ea97-4f56-9553-9e1ea84c10ee","Type":"ContainerDied","Data":"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4"} Nov 01 22:04:24.362997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.360781 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" event={"ID":"359f99ba-ea97-4f56-9553-9e1ea84c10ee","Type":"ContainerDied","Data":"afce06068c7f9a11d32495e893be664ff459cb17cedcd0cdf6c7421d087ab3c4"} Nov 01 22:04:24.362997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.360848 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.372927 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.372950 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.372963 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.372977 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.372990 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.373004 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.373017 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.373052 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xpv\" (UniqueName: \"kubernetes.io/projected/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-kube-api-access-s2xpv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.373306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.373065 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.377084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.376978 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" podStartSLOduration=1.924365435 podStartE2EDuration="2.376963156s" totalImagesPullingTime="452.597721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:24.376513909 +0000 UTC m=+1211.350416448" watchObservedRunningTime="2026-11-01 22:04:24.376963156 +0000 UTC m=+1211.350865695" Nov 01 22:04:24.378158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.378138 4395 scope.go:122] "RemoveContainer" containerID="2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671" Nov 01 22:04:24.378553 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:24.378525 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671\": container with ID starting with 2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671 not found: ID does not exist" containerID="2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671" Nov 01 22:04:24.378660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.378568 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671"} err="failed to get container status \"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671\": rpc error: code = NotFound desc = could not find container \"2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671\": container with ID starting with 2990689290c9b51e9e804dbf54bc52df2a5c3c44e8d73218e9928630fe863671 not found: ID does not exist" Nov 01 22:04:24.378660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.378597 4395 scope.go:122] "RemoveContainer" containerID="4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e" Nov 01 22:04:24.394908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.392129 4395 scope.go:122] "RemoveContainer" containerID="4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e" Nov 01 22:04:24.394908 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:24.392435 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e\": container with ID starting with 4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e not found: ID does not exist" containerID="4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e" Nov 01 22:04:24.394908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.392469 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e"} err="failed to get container status \"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e\": rpc error: code = NotFound desc = could not find container \"4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e\": container with ID starting with 4b96e0204ff64d119f6399e1a9e6360aceb05c1d0aba29a507a7aec9fd698a3e not found: ID does not exist" Nov 01 22:04:24.394908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.392505 4395 scope.go:122] "RemoveContainer" containerID="dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4" Nov 01 22:04:24.410578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.410534 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 22:04:24.421166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.421148 4395 scope.go:122] "RemoveContainer" containerID="dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4" Nov 01 22:04:24.421444 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:24.421427 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4\": container with ID starting with dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4 not found: ID does not exist" containerID="dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4" Nov 01 22:04:24.421521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.421451 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4"} err="failed to get container status \"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4\": rpc error: code = NotFound desc = could not find container \"dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4\": container with ID starting with dda1d12b94788f9a10375faf3a05261bbe787da46b16fa66f8608e2ae771eab4 not found: ID does not exist" Nov 01 22:04:24.422908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.422880 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-6sm24"] Nov 01 22:04:24.435003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.434983 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 22:04:24.435143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.435084 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 22:04:24.438086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.438063 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-ph8sb"] Nov 01 22:04:24.486044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.485993 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 22:04:24.491572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.491546 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-l2c57"] Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.585614 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.585747 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.585834 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.585894 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.585973 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.586040 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.586110 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.586181 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.593993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.586266 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\") pod \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\" (UID: \"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0\") " Nov 01 22:04:24.594641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.594613 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.611257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.595348 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:24.611257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.605117 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.611257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.605255 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.611681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.611645 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.652056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.648493 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.653454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.653283 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:04:24.653454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.653403 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:04:24.653666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.653518 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw" pod "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" (UID: "9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0"). InnerVolumeSpecName "kube-api-access-57zsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:24.689385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689287 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689566 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57zsw\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-kube-api-access-57zsw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689596 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689610 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689834 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689855 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689868 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.689906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.689881 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:24.690242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:24.690214 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:25.167256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.167217 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" path="/var/lib/kubelet/pods/359f99ba-ea97-4f56-9553-9e1ea84c10ee/volumes" Nov 01 22:04:25.168054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.168016 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" path="/var/lib/kubelet/pods/6529c812-e3d7-46b9-9320-05958ff0580b/volumes" Nov 01 22:04:25.168563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.168543 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" path="/var/lib/kubelet/pods/abaa57b6-8fa5-4290-b9b0-e7f3b9b60293/volumes" Nov 01 22:04:25.394650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.394615 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" Nov 01 22:04:25.394650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.394626 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts" event={"ID":"9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0","Type":"ContainerDied","Data":"d0005797060fced441bd19c333976aae46b2f8c96c87725b8bc8e1dc716368c6"} Nov 01 22:04:25.394914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.394697 4395 scope.go:122] "RemoveContainer" containerID="b960aeb53472fb592840b1859c83604a39306e207d051a85bd5388763e290380" Nov 01 22:04:25.418382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.418325 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 22:04:25.425718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:25.425688 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-4zzts"] Nov 01 22:04:26.738664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.738627 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:04:26.739352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739336 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" Nov 01 22:04:26.739405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739352 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" Nov 01 22:04:26.739405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739375 4395 cpu_manager.go:414] "containerMap: removing container" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" Nov 01 22:04:26.739405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739380 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" Nov 01 22:04:26.739405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739396 4395 cpu_manager.go:414] "containerMap: removing container" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" Nov 01 22:04:26.739405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739403 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739433 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739438 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739533 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739544 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6529c812-e3d7-46b9-9320-05958ff0580b" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739552 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="359f99ba-ea97-4f56-9553-9e1ea84c10ee" containerName="istio-proxy" Nov 01 22:04:26.739608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.739560 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="abaa57b6-8fa5-4290-b9b0-e7f3b9b60293" containerName="istio-proxy" Nov 01 22:04:26.744695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.744675 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:26.751601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.751571 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:04:26.751740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.751726 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0390e1\"" Nov 01 22:04:26.769082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.769021 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:04:26.774733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.774713 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:26.786325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.786302 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:04:26.848659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.848583 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0390e1-dockercfg-qs8cx\"" Nov 01 22:04:26.873968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.855047 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:26.875512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.875482 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr5r\" (UniqueName: \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:26.894257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.894219 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:26.894611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.894585 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msfw\" (UniqueName: \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.999108 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr5r\" (UniqueName: \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.999177 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.999238 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msfw\" (UniqueName: \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:26.999344 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.000587 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:27.002105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.001818 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.042053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.041488 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msfw\" (UniqueName: \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\") pod \"payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.077048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.076851 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr5r\" (UniqueName: \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\") pod \"payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:27.113055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.099311 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:27.194948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.194080 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0" path="/var/lib/kubelet/pods/9c72e686-3ea0-40f7-98b5-5b1ba24eb5c0/volumes" Nov 01 22:04:27.372919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.372867 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:27.510782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.510750 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:04:27.515090 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:27.515061 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2d293b_459f_4f60_9964_c967b95a51e8.slice/crio-2b7293bc1787ce95a51858e055ac43d4f0d4f91563e64ebc1532874b4c3ab467 WatchSource:0}: Error finding container 2b7293bc1787ce95a51858e055ac43d4f0d4f91563e64ebc1532874b4c3ab467: Status 404 returned error can't find the container with id 2b7293bc1787ce95a51858e055ac43d4f0d4f91563e64ebc1532874b4c3ab467 Nov 01 22:04:27.665863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:27.665836 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:04:27.666363 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:27.666342 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2207a59f_da22_43f1_9b51_4d040376d962.slice/crio-ab6ec5703a1a6563846a016c2c09f6c09434ef0d7b4a7618cc46b06528ae8d20 WatchSource:0}: Error finding container ab6ec5703a1a6563846a016c2c09f6c09434ef0d7b4a7618cc46b06528ae8d20: Status 404 returned error can't find the container with id ab6ec5703a1a6563846a016c2c09f6c09434ef0d7b4a7618cc46b06528ae8d20 Nov 01 22:04:28.040313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.040279 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:04:28.044911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.044892 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.050774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.050749 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-21dd16de-dockercfg-v6rw8\"" Nov 01 22:04:28.059069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.058109 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:04:28.088924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.088894 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:04:28.093858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.093837 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.107167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.107142 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:04:28.124423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.124387 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhq6j\" (UniqueName: \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.124602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.124525 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.155051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.145398 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-21dd16de\"" Nov 01 22:04:28.304120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.269187 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhq6j\" (UniqueName: \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.304120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.269347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.304120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.269391 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.304120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.269518 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6z2x\" (UniqueName: \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.304120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.271084 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.386092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.380360 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.394053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.390259 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6z2x\" (UniqueName: \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.458051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.456711 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhq6j\" (UniqueName: \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.458051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.457303 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.534062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.529747 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6z2x\" (UniqueName: \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\") pod \"payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:28.538079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.536882 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" event={"ID":"2207a59f-da22-43f1-9b51-4d040376d962","Type":"ContainerStarted","Data":"9fd6a6d80dc48090970ec516084d340b2c3d6d330b14a34f79ec911530a4efcb"} Nov 01 22:04:28.538079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.536966 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" event={"ID":"2207a59f-da22-43f1-9b51-4d040376d962","Type":"ContainerStarted","Data":"ab6ec5703a1a6563846a016c2c09f6c09434ef0d7b4a7618cc46b06528ae8d20"} Nov 01 22:04:28.538285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.538139 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:28.540912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.540883 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" event={"ID":"5c2d293b-459f-4f60-9964-c967b95a51e8","Type":"ContainerStarted","Data":"a4401a2a7ecfcf1d5b05e63b385b5ccc5e51c8a0188818bac7d4b210d8c53d1b"} Nov 01 22:04:28.541093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.540932 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" event={"ID":"5c2d293b-459f-4f60-9964-c967b95a51e8","Type":"ContainerStarted","Data":"2b7293bc1787ce95a51858e055ac43d4f0d4f91563e64ebc1532874b4c3ab467"} Nov 01 22:04:28.541160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.541112 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:28.563297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.563203 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" podStartSLOduration=2.563186568 podStartE2EDuration="2.563186568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:28.557613884 +0000 UTC m=+1215.531516424" watchObservedRunningTime="2026-11-01 22:04:28.563186568 +0000 UTC m=+1215.537089110" Nov 01 22:04:28.576441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.576394 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" podStartSLOduration=2.576386252 podStartE2EDuration="2.576386252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:28.574769248 +0000 UTC m=+1215.548671789" watchObservedRunningTime="2026-11-01 22:04:28.576386252 +0000 UTC m=+1215.550288791" Nov 01 22:04:28.677659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.677633 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:28.726562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:28.726529 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:29.090580 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.090554 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:04:29.090990 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:29.090554 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b4637b_d75b_4afd_be6e_92c3bdfcabbb.slice/crio-93f61323a9eb5bcbd903d9db4612124c72c33ebedb5b1b9ef8d1aa9e9f8d1770 WatchSource:0}: Error finding container 93f61323a9eb5bcbd903d9db4612124c72c33ebedb5b1b9ef8d1aa9e9f8d1770: Status 404 returned error can't find the container with id 93f61323a9eb5bcbd903d9db4612124c72c33ebedb5b1b9ef8d1aa9e9f8d1770 Nov 01 22:04:29.098480 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.096806 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:04:29.549985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.549954 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" event={"ID":"f5140852-5478-4659-a65e-ae2c74629f33","Type":"ContainerStarted","Data":"0a20f8e4abf95ad21b2883cd4d6b372231a202f150d91b9ff79ba1f490e28938"} Nov 01 22:04:29.550267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.550011 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" event={"ID":"f5140852-5478-4659-a65e-ae2c74629f33","Type":"ContainerStarted","Data":"c57c2bd9c384229d5740ad42b78010ac1c1aed6c8556fc90f12d24e1778e68cc"} Nov 01 22:04:29.550267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.550084 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:29.551676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.551652 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" event={"ID":"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb","Type":"ContainerStarted","Data":"20ded8dca308fd97c90034ce2f93606eef83a0653460f902bd0d78482b79e75b"} Nov 01 22:04:29.551798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.551694 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" event={"ID":"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb","Type":"ContainerStarted","Data":"93f61323a9eb5bcbd903d9db4612124c72c33ebedb5b1b9ef8d1aa9e9f8d1770"} Nov 01 22:04:29.551798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.551744 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:29.571674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.571618 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" podStartSLOduration=1.571601314 podStartE2EDuration="1.571601314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:29.56919477 +0000 UTC m=+1216.543097308" watchObservedRunningTime="2026-11-01 22:04:29.571601314 +0000 UTC m=+1216.545503854" Nov 01 22:04:29.598961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:29.598894 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" podStartSLOduration=1.598876469 podStartE2EDuration="1.598876469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:29.589255727 +0000 UTC m=+1216.563158269" watchObservedRunningTime="2026-11-01 22:04:29.598876469 +0000 UTC m=+1216.572779009" Nov 01 22:04:30.419562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:30.419529 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:04:32.377475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.377436 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:04:32.383775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.383751 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.394782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.394695 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:04:32.396485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.396456 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-004180-serving-cert\"" Nov 01 22:04:32.473243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.473214 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.473410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.473253 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.473410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.473315 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.473518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.473432 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnpw\" (UniqueName: \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.588757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.588729 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.590530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.590487 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.590697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.590540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.590697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.590675 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnpw\" (UniqueName: \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.591196 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:32.588891 4395 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-004180-serving-cert: secret "maas-api-e2e-api-b-004180-serving-cert" not found Nov 01 22:04:32.591345 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:32.591248 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls podName:2d9afd5e-0e61-4b1b-9326-d1c86e4a8410 nodeName:}" failed. No retries permitted until 2026-11-01 22:04:33.091225794 +0000 UTC m=+1220.065128325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls") pod "maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" (UID: "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410") : secret "maas-api-e2e-api-b-004180-serving-cert" not found Nov 01 22:04:32.592567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.592544 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.595868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.595847 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:32.615792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:32.615724 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnpw\" (UniqueName: \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:33.107352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:33.107310 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:33.117653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:33.114795 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") pod \"maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:33.313511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:33.313478 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:33.609044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:33.609000 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:04:33.613560 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:33.613529 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9afd5e_0e61_4b1b_9326_d1c86e4a8410.slice/crio-9def4b90fbfdd54b9306e758e529af2d187927744e541492bc0cb306764691cf WatchSource:0}: Error finding container 9def4b90fbfdd54b9306e758e529af2d187927744e541492bc0cb306764691cf: Status 404 returned error can't find the container with id 9def4b90fbfdd54b9306e758e529af2d187927744e541492bc0cb306764691cf Nov 01 22:04:34.608485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:34.608443 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" event={"ID":"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410","Type":"ContainerStarted","Data":"9c3982c2eec3ad606e25671e0de3affe253fede0d326e02dc5d98c0d3dbede27"} Nov 01 22:04:34.608661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:34.608547 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:34.608661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:34.608568 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" event={"ID":"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410","Type":"ContainerStarted","Data":"9def4b90fbfdd54b9306e758e529af2d187927744e541492bc0cb306764691cf"} Nov 01 22:04:34.629241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:34.629194 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" podStartSLOduration=2.186528807 podStartE2EDuration="2.629180352s" totalImagesPullingTime="442.651545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:34.627616356 +0000 UTC m=+1221.601518895" watchObservedRunningTime="2026-11-01 22:04:34.629180352 +0000 UTC m=+1221.603082961" Nov 01 22:04:37.156556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.156528 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:04:37.174461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.174432 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.180550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.180523 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-21dd16de-serving-cert\"" Nov 01 22:04:37.196205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.196178 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:04:37.253394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.253367 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.253532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.253414 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.253532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.253509 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqrb\" (UniqueName: \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.253621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.253568 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.355947 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.356454 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.356537 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqrb\" (UniqueName: \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.356603 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:37.356849 4395 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-serving-cert: secret "maas-api-e2e-mt-21dd16de-serving-cert" not found Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:37.356938 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls podName:422bea6e-d4ef-426a-9343-825cb0c93269 nodeName:}" failed. No retries permitted until 2026-11-01 22:04:37.856908298 +0000 UTC m=+1224.830810831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls") pod "maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" (UID: "422bea6e-d4ef-426a-9343-825cb0c93269") : secret "maas-api-e2e-mt-21dd16de-serving-cert" not found Nov 01 22:04:37.363569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.358187 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.397119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.381642 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.397119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.394000 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqrb\" (UniqueName: \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.870176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.866459 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:37.904053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:37.901623 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:38.105556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:38.105517 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:38.411846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:38.411816 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:04:38.412389 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:38.412366 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422bea6e_d4ef_426a_9343_825cb0c93269.slice/crio-5637f0785c54c73b2126423e0d8dae23a78995f32c112861596b8605e37b0d31 WatchSource:0}: Error finding container 5637f0785c54c73b2126423e0d8dae23a78995f32c112861596b8605e37b0d31: Status 404 returned error can't find the container with id 5637f0785c54c73b2126423e0d8dae23a78995f32c112861596b8605e37b0d31 Nov 01 22:04:38.657075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:38.657009 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" event={"ID":"422bea6e-d4ef-426a-9343-825cb0c93269","Type":"ContainerStarted","Data":"5637f0785c54c73b2126423e0d8dae23a78995f32c112861596b8605e37b0d31"} Nov 01 22:04:39.651056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:39.628570 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:04:39.878152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:39.864846 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" event={"ID":"422bea6e-d4ef-426a-9343-825cb0c93269","Type":"ContainerStarted","Data":"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589"} Nov 01 22:04:39.879653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:39.879324 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:40.640977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:40.640944 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:04:40.659216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:40.659185 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:04:40.659881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:40.659837 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:04:40.660706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:40.660685 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:04:40.699077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:40.698822 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" podStartSLOduration=3.317708665 podStartE2EDuration="3.698801205s" totalImagesPullingTime="381.09254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:40.058383392 +0000 UTC m=+1227.032285957" watchObservedRunningTime="2026-11-01 22:04:40.698801205 +0000 UTC m=+1227.672703737" Nov 01 22:04:41.557361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.557330 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:04:41.562116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.562092 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.567303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.567272 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0390e1-serving-cert\"" Nov 01 22:04:41.579310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.579282 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:04:41.707836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.707334 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.708634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.708577 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.708903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.708885 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.709246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.709227 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8mk\" (UniqueName: \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.817110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.811133 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.817110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.811347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8mk\" (UniqueName: \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.817110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.811468 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.817110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.811539 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.817110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.812508 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.826583 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.826548 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.833764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.833720 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.850447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.850340 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8mk\" (UniqueName: \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\") pod \"maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:41.887979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:41.887943 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:42.192004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:42.191975 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:04:42.192302 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:04:42.192279 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8c9693_c252_413c_a0e9_ba452ef8a42a.slice/crio-27dfe02e55fad896ca653bdbf7d969fab21f519d1c6f38e631d3573562ae196d WatchSource:0}: Error finding container 27dfe02e55fad896ca653bdbf7d969fab21f519d1c6f38e631d3573562ae196d: Status 404 returned error can't find the container with id 27dfe02e55fad896ca653bdbf7d969fab21f519d1c6f38e631d3573562ae196d Nov 01 22:04:42.905693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:42.905661 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" event={"ID":"8a8c9693-c252-413c-a0e9-ba452ef8a42a","Type":"ContainerStarted","Data":"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d"} Nov 01 22:04:42.906086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:42.905714 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" event={"ID":"8a8c9693-c252-413c-a0e9-ba452ef8a42a","Type":"ContainerStarted","Data":"27dfe02e55fad896ca653bdbf7d969fab21f519d1c6f38e631d3573562ae196d"} Nov 01 22:04:42.906086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:42.905874 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:42.929664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:42.929595 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" podStartSLOduration=1.4179580729999999 podStartE2EDuration="1.929580144s" totalImagesPullingTime="511.622071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:42.926301757 +0000 UTC m=+1229.900204300" watchObservedRunningTime="2026-11-01 22:04:42.929580144 +0000 UTC m=+1229.903482714" Nov 01 22:04:44.981052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:44.978050 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:04:44.981052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:44.978441 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:04:45.098062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:45.095214 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.151381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:45.151319 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.264111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:45.260648 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:45.359424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:45.359375 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:04:47.046093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.041164 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:04:47.385615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.385141 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:47.385615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.385380 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:04:47.434875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.434837 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:04:47.435548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.435134 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:47.556918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:47.527882 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.016054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.015472 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.041054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.040214 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.112060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.077617 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:04:48.125964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.084684 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.133852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.133822 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:04:48.134170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.134149 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.141414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.141392 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-702002\"" Nov 01 22:04:48.141837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.141804 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-702002-dockercfg-bg6pq\"" Nov 01 22:04:48.147366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.147330 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 22:04:48.165189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.165149 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.216056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.192584 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.217123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.217091 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vqp\" (UniqueName: \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.333126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.332231 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vqp\" (UniqueName: \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.333126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.332443 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.337053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.333496 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.449249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.449194 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vqp\" (UniqueName: \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\") pod \"payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:48.450181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.450140 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:48.451082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.451060 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:04:48.453367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.453343 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:04:48.454014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.453994 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:04:48.811590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:48.668974 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:49.095101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.092215 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:04:49.110060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.105419 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:04:49.141074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.122291 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:49.142285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.142255 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:04:49.150847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.150810 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:49.151499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.151475 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:04:49.196588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.193630 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:04:49.212543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.210445 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:04:49.464207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.212921 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:04:49.465233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.465207 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:04:49.465477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.465462 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:04:49.477264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.477234 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:04:49.569479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.569439 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:49.626098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.626046 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:04:49.629514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.629485 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:04:49.700109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:49.689217 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 01 22:04:50.184336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.184291 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:50.758830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.758780 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 01 22:04:50.759086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.262020 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:50.759487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.276195 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:04:50.772091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.502549 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:50.781210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.193134 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:04:50.788479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.784945 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:50.788708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.788682 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:04:50.934252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.933810 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:04:50.974634 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:50.954470 4395 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.358s" Nov 01 22:04:50.974634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.954567 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:04:50.998233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.974804 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:04:50.998233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:50.978193 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk" podUID="1429c5b6-feda-4fa4-8fd9-f041fe7f98ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Nov 01 22:04:51.042622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.042534 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:04:51.114068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.083014 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:04:51.114068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.096389 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:04:51.128873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.128842 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:04:51.196458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.196413 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2mhpk" podUID="3febf414-14ee-4545-abfa-1f421ad43f31" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:51.235488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.235414 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-svbr7" podUID="79984a24-8785-4acc-8259-cc1e4f908fb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:51.317739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.317659 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:04:51.328165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.328135 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:04:51.333247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.333200 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" event={"ID":"ba346d51-88f9-468a-817e-d5963a8165e2","Type":"ContainerStarted","Data":"fc3a11d045098675689eca975e60ad8731926589a880b01579c3b9f9dbd26480"} Nov 01 22:04:51.351444 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.351367 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:04:51.459735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:51.459709 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:04:52.329675 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:52.329643 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:04:52.342233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:52.342199 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" event={"ID":"ba346d51-88f9-468a-817e-d5963a8165e2","Type":"ContainerStarted","Data":"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc"} Nov 01 22:04:52.342410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:52.342288 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:04:52.362501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:52.362439 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" podStartSLOduration=5.3624203 podStartE2EDuration="5.3624203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:52.36212401 +0000 UTC m=+1239.336026544" watchObservedRunningTime="2026-11-01 22:04:52.3624203 +0000 UTC m=+1239.336322840" Nov 01 22:04:56.485057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:56.481939 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" podUID="60920858-f25c-4d5c-9e5d-dda600494577" containerName="payload-processing" containerID="cri-o://ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843" gracePeriod=25 Nov 01 22:04:57.071161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.071136 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:04:57.246539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.246466 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\" (UniqueName: \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\") pod \"60920858-f25c-4d5c-9e5d-dda600494577\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " Nov 01 22:04:57.246700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.246534 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\") pod \"60920858-f25c-4d5c-9e5d-dda600494577\" (UID: \"60920858-f25c-4d5c-9e5d-dda600494577\") " Nov 01 22:04:57.246938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.246912 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume" pod "60920858-f25c-4d5c-9e5d-dda600494577" (UID: "60920858-f25c-4d5c-9e5d-dda600494577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:04:57.247159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.247139 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60920858-f25c-4d5c-9e5d-dda600494577-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:57.248775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.248750 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w" pod "60920858-f25c-4d5c-9e5d-dda600494577" (UID: "60920858-f25c-4d5c-9e5d-dda600494577"). InnerVolumeSpecName "kube-api-access-zln5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:04:57.348624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.348593 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zln5w\" (UniqueName: \"kubernetes.io/projected/60920858-f25c-4d5c-9e5d-dda600494577-kube-api-access-zln5w\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:04:57.471415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.471377 4395 generic.go:356] "Generic (PLEG): container finished" podID="60920858-f25c-4d5c-9e5d-dda600494577" containerID="ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843" exitCode=0 Nov 01 22:04:57.471415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.471418 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" Nov 01 22:04:57.471658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.471457 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" event={"ID":"60920858-f25c-4d5c-9e5d-dda600494577","Type":"ContainerDied","Data":"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843"} Nov 01 22:04:57.471658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.471517 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66" event={"ID":"60920858-f25c-4d5c-9e5d-dda600494577","Type":"ContainerDied","Data":"2a6b81ce74dd0f7cedeab2a66e878e10604200f87675fcee57eabe8e29f734ba"} Nov 01 22:04:57.471658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.471546 4395 scope.go:122] "RemoveContainer" containerID="ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843" Nov 01 22:04:57.485245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.485195 4395 scope.go:122] "RemoveContainer" containerID="ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843" Nov 01 22:04:57.486288 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:04:57.485781 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843\": container with ID starting with ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843 not found: ID does not exist" containerID="ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843" Nov 01 22:04:57.486288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.485818 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843"} err="failed to get container status \"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843\": rpc error: code = NotFound desc = could not find container \"ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843\": container with ID starting with ef6dc34872b0b729fe77b8911500bc01ab5b30596868a6f831469d94f3e4a843 not found: ID does not exist" Nov 01 22:04:57.504397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.504206 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:04:57.508544 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:57.508524 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b1b5598d-684b864db8-46b66"] Nov 01 22:04:58.481546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.481514 4395 generic.go:356] "Generic (PLEG): container finished" podID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerID="4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c" exitCode=0 Nov 01 22:04:58.481799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.481598 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerDied","Data":"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c"} Nov 01 22:04:58.482102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.482085 4395 scope.go:122] "RemoveContainer" containerID="4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c" Nov 01 22:04:58.483872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.483847 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:04:58.483942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.483918 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:04:58.483978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:58.483949 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:04:59.221051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:59.216124 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60920858-f25c-4d5c-9e5d-dda600494577" path="/var/lib/kubelet/pods/60920858-f25c-4d5c-9e5d-dda600494577/volumes" Nov 01 22:04:59.507152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:59.506308 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerStarted","Data":"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234"} Nov 01 22:04:59.573523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:59.573492 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:04:59.592077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:59.583147 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:04:59.592077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:04:59.583227 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:00.578051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:00.575150 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:05:00.578051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:00.575225 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:01.573792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:01.573757 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:05:01.573958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:01.573821 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:02.575054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:02.574467 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:05:02.575054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:02.574537 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:03.369697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:03.369663 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:05:03.573349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:03.573314 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:05:03.573524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:03.573376 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:03.955128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:03.954993 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:05:04.573438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:04.573406 4395 patch_prober.go:29] interesting pod/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:05:04.573617 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:04.573472 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 01 22:05:05.598274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:05.598245 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:05:05.600460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:05.600435 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:05:13.426443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.426413 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:05:13.427301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.427283 4395 cpu_manager.go:414] "containerMap: removing container" podUID="60920858-f25c-4d5c-9e5d-dda600494577" containerName="payload-processing" Nov 01 22:05:13.427301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.427301 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60920858-f25c-4d5c-9e5d-dda600494577" containerName="payload-processing" Nov 01 22:05:13.427427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.427406 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="60920858-f25c-4d5c-9e5d-dda600494577" containerName="payload-processing" Nov 01 22:05:13.432916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.432895 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.440549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.440524 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:05:13.443018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.442996 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bb882c-dockercfg-xgn45\"" Nov 01 22:05:13.482514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.482477 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2z7\" (UniqueName: \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.482703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.482683 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.508384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.508337 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bb882c\"" Nov 01 22:05:13.584168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.584135 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2z7\" (UniqueName: \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.585302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.585269 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.594539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.586535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.607758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.607725 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2z7\" (UniqueName: \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\") pod \"payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:13.760444 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:13.760406 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:14.166515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.166416 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:05:14.184501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.184462 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:05:14.247646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.247619 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:05:14.725054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.715018 4395 scope.go:122] "RemoveContainer" containerID="251e65fe730c0d5aaad54334a3bb5faaa304695ccd8f7a513af0616eb719ca48" Nov 01 22:05:14.774939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.774902 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" event={"ID":"5014d6cb-8d99-4691-932f-f8c0ca090cea","Type":"ContainerStarted","Data":"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838"} Nov 01 22:05:14.775090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.774967 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" event={"ID":"5014d6cb-8d99-4691-932f-f8c0ca090cea","Type":"ContainerStarted","Data":"9337ce6401367a5a59e1e0489ee2924419a89f8c1e8dbfb11a40c4acec2ab399"} Nov 01 22:05:14.775287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.775267 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:14.799492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:14.799444 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" podStartSLOduration=1.799429146 podStartE2EDuration="1.799429146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:14.794377973 +0000 UTC m=+1261.768280522" watchObservedRunningTime="2026-11-01 22:05:14.799429146 +0000 UTC m=+1261.773331702" Nov 01 22:05:18.590270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:18.590238 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:05:18.602293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:18.602266 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:05:19.170599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.167941 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" podUID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" containerName="payload-processing" containerID="cri-o://20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31" gracePeriod=25 Nov 01 22:05:19.191966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.191930 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" podUID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" containerName="payload-pre-processing" containerID="cri-o://2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9" gracePeriod=25 Nov 01 22:05:19.555902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.550495 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:05:19.623052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.615628 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\" (UniqueName: \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\") pod \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " Nov 01 22:05:19.623052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.615764 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\") pod \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\" (UID: \"596e5fb4-9f8d-44a7-926e-bf8995c60e52\") " Nov 01 22:05:19.623052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.619979 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume" pod "596e5fb4-9f8d-44a7-926e-bf8995c60e52" (UID: "596e5fb4-9f8d-44a7-926e-bf8995c60e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:19.682211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.682172 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s" pod "596e5fb4-9f8d-44a7-926e-bf8995c60e52" (UID: "596e5fb4-9f8d-44a7-926e-bf8995c60e52"). InnerVolumeSpecName "kube-api-access-9h82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:19.687423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.687394 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:05:19.718673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.718640 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h82s\" (UniqueName: \"kubernetes.io/projected/596e5fb4-9f8d-44a7-926e-bf8995c60e52-kube-api-access-9h82s\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:19.718673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.718670 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596e5fb4-9f8d-44a7-926e-bf8995c60e52-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:19.813612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.813580 4395 generic.go:356] "Generic (PLEG): container finished" podID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" containerID="2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9" exitCode=0 Nov 01 22:05:19.813786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.813625 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" Nov 01 22:05:19.813786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.813669 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" event={"ID":"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0","Type":"ContainerDied","Data":"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9"} Nov 01 22:05:19.813786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.813751 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5" event={"ID":"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0","Type":"ContainerDied","Data":"41c0b86e03c5e7efbb4350d24ee4c31968fe0751b996ba926007c7ca56645ad9"} Nov 01 22:05:19.813786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.813781 4395 scope.go:122] "RemoveContainer" containerID="2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9" Nov 01 22:05:19.815194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.815170 4395 generic.go:356] "Generic (PLEG): container finished" podID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" containerID="20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31" exitCode=0 Nov 01 22:05:19.815330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.815206 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" event={"ID":"596e5fb4-9f8d-44a7-926e-bf8995c60e52","Type":"ContainerDied","Data":"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31"} Nov 01 22:05:19.815330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.815221 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" Nov 01 22:05:19.815330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.815256 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw" event={"ID":"596e5fb4-9f8d-44a7-926e-bf8995c60e52","Type":"ContainerDied","Data":"b12e838b2560f38f7d17089d96f6504f4eaa2b682e8843c32745197cba4a0b06"} Nov 01 22:05:19.819314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.819296 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\") pod \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " Nov 01 22:05:19.819426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.819384 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\" (UniqueName: \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\") pod \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\" (UID: \"830982df-60fe-4f5b-a6a7-d2b1da6dc5a0\") " Nov 01 22:05:19.819665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.819644 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume" pod "830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" (UID: "830982df-60fe-4f5b-a6a7-d2b1da6dc5a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:19.819910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.819892 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:19.821658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.821638 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4" pod "830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" (UID: "830982df-60fe-4f5b-a6a7-d2b1da6dc5a0"). InnerVolumeSpecName "kube-api-access-9qtb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:19.825134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.825112 4395 scope.go:122] "RemoveContainer" containerID="2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9" Nov 01 22:05:19.825382 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:19.825359 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9\": container with ID starting with 2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9 not found: ID does not exist" containerID="2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9" Nov 01 22:05:19.825426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.825391 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9"} err="failed to get container status \"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9\": rpc error: code = NotFound desc = could not find container \"2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9\": container with ID starting with 2aba518439d509115f6b99fc4eb49df73cae701a15cde141d100f51e1b563dc9 not found: ID does not exist" Nov 01 22:05:19.825426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.825407 4395 scope.go:122] "RemoveContainer" containerID="20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31" Nov 01 22:05:19.834915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.834902 4395 scope.go:122] "RemoveContainer" containerID="20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31" Nov 01 22:05:19.835171 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:19.835155 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31\": container with ID starting with 20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31 not found: ID does not exist" containerID="20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31" Nov 01 22:05:19.835225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.835178 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31"} err="failed to get container status \"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31\": rpc error: code = NotFound desc = could not find container \"20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31\": container with ID starting with 20bb56d5fa42ebd94aa1f09fde93d7a14ee16241292cfee1d5685a783a9bea31 not found: ID does not exist" Nov 01 22:05:19.840728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.840711 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:05:19.844847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.844827 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d506c6-64b85f4d8b-5xsmw"] Nov 01 22:05:19.921552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:19.921531 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtb4\" (UniqueName: \"kubernetes.io/projected/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0-kube-api-access-9qtb4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:20.138127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:20.138089 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:05:20.147250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:20.147222 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d506c6-9698d4dc-46cl5"] Nov 01 22:05:20.808212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:20.808177 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:05:20.808713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:20.808427 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" podUID="422bea6e-d4ef-426a-9343-825cb0c93269" containerName="maas-api" containerID="cri-o://be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589" gracePeriod=30 Nov 01 22:05:21.467935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.459638 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" path="/var/lib/kubelet/pods/596e5fb4-9f8d-44a7-926e-bf8995c60e52/volumes" Nov 01 22:05:21.467935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.460601 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" path="/var/lib/kubelet/pods/830982df-60fe-4f5b-a6a7-d2b1da6dc5a0/volumes" Nov 01 22:05:21.588682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.588650 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:05:21.666062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.665997 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") pod \"422bea6e-d4ef-426a-9343-825cb0c93269\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " Nov 01 22:05:21.666358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.666334 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\") pod \"422bea6e-d4ef-426a-9343-825cb0c93269\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " Nov 01 22:05:21.666682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.666659 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\" (UniqueName: \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\") pod \"422bea6e-d4ef-426a-9343-825cb0c93269\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " Nov 01 22:05:21.666777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.666703 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca" pod "422bea6e-d4ef-426a-9343-825cb0c93269" (UID: "422bea6e-d4ef-426a-9343-825cb0c93269"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:21.666866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.666852 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\") pod \"422bea6e-d4ef-426a-9343-825cb0c93269\" (UID: \"422bea6e-d4ef-426a-9343-825cb0c93269\") " Nov 01 22:05:21.667488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.667466 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/422bea6e-d4ef-426a-9343-825cb0c93269-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:21.669805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.669784 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs" pod "422bea6e-d4ef-426a-9343-825cb0c93269" (UID: "422bea6e-d4ef-426a-9343-825cb0c93269"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:21.671407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.671387 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb" pod "422bea6e-d4ef-426a-9343-825cb0c93269" (UID: "422bea6e-d4ef-426a-9343-825cb0c93269"). InnerVolumeSpecName "kube-api-access-sjqrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:21.672712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.672694 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls" pod "422bea6e-d4ef-426a-9343-825cb0c93269" (UID: "422bea6e-d4ef-426a-9343-825cb0c93269"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:21.769148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.769118 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqrb\" (UniqueName: \"kubernetes.io/projected/422bea6e-d4ef-426a-9343-825cb0c93269-kube-api-access-sjqrb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:21.769148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.769147 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:21.769349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.769156 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/422bea6e-d4ef-426a-9343-825cb0c93269-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:21.839648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.839619 4395 generic.go:356] "Generic (PLEG): container finished" podID="422bea6e-d4ef-426a-9343-825cb0c93269" containerID="be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589" exitCode=0 Nov 01 22:05:21.840141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.839666 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" event={"ID":"422bea6e-d4ef-426a-9343-825cb0c93269","Type":"ContainerDied","Data":"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589"} Nov 01 22:05:21.840141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.839670 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" Nov 01 22:05:21.840141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.839709 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c" event={"ID":"422bea6e-d4ef-426a-9343-825cb0c93269","Type":"ContainerDied","Data":"5637f0785c54c73b2126423e0d8dae23a78995f32c112861596b8605e37b0d31"} Nov 01 22:05:21.840141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.839733 4395 scope.go:122] "RemoveContainer" containerID="be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589" Nov 01 22:05:21.853910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.853888 4395 scope.go:122] "RemoveContainer" containerID="be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589" Nov 01 22:05:21.854293 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:21.854264 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589\": container with ID starting with be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589 not found: ID does not exist" containerID="be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589" Nov 01 22:05:21.854392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.854300 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589"} err="failed to get container status \"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589\": rpc error: code = NotFound desc = could not find container \"be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589\": container with ID starting with be1d979318d6d4ed34804f0d6939a8b16585f13394fadf1243e43dfde44df589 not found: ID does not exist" Nov 01 22:05:21.869438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.869412 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:05:21.873611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:21.873588 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-4fk7c"] Nov 01 22:05:22.449117 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:22.447753 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:05:23.167427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.167395 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422bea6e-d4ef-426a-9343-825cb0c93269" path="/var/lib/kubelet/pods/422bea6e-d4ef-426a-9343-825cb0c93269/volumes" Nov 01 22:05:23.590604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.590559 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" podUID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" containerName="payload-processing" containerID="cri-o://20ded8dca308fd97c90034ce2f93606eef83a0653460f902bd0d78482b79e75b" gracePeriod=25 Nov 01 22:05:23.603002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.602964 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" podUID="f5140852-5478-4659-a65e-ae2c74629f33" containerName="payload-pre-processing" containerID="cri-o://0a20f8e4abf95ad21b2883cd4d6b372231a202f150d91b9ff79ba1f490e28938" gracePeriod=25 Nov 01 22:05:23.914968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.914930 4395 generic.go:356] "Generic (PLEG): container finished" podID="f5140852-5478-4659-a65e-ae2c74629f33" containerID="0a20f8e4abf95ad21b2883cd4d6b372231a202f150d91b9ff79ba1f490e28938" exitCode=0 Nov 01 22:05:23.915182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.915087 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" event={"ID":"f5140852-5478-4659-a65e-ae2c74629f33","Type":"ContainerDied","Data":"0a20f8e4abf95ad21b2883cd4d6b372231a202f150d91b9ff79ba1f490e28938"} Nov 01 22:05:23.919918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.919883 4395 generic.go:356] "Generic (PLEG): container finished" podID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" containerID="20ded8dca308fd97c90034ce2f93606eef83a0653460f902bd0d78482b79e75b" exitCode=0 Nov 01 22:05:23.920159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.920126 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" event={"ID":"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb","Type":"ContainerDied","Data":"20ded8dca308fd97c90034ce2f93606eef83a0653460f902bd0d78482b79e75b"} Nov 01 22:05:23.955807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:23.955776 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:05:24.006907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.006880 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:05:24.040643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.040605 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\") pod \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " Nov 01 22:05:24.040851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.040710 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\" (UniqueName: \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\") pod \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\" (UID: \"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb\") " Nov 01 22:05:24.041042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.040996 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume" pod "d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" (UID: "d0b4637b-d75b-4afd-be6e-92c3bdfcabbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:24.041540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.041503 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:24.043225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.043193 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x" pod "d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" (UID: "d0b4637b-d75b-4afd-be6e-92c3bdfcabbb"). InnerVolumeSpecName "kube-api-access-l6z2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:24.141950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.141921 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\" (UniqueName: \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\") pod \"f5140852-5478-4659-a65e-ae2c74629f33\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " Nov 01 22:05:24.142167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.141982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\") pod \"f5140852-5478-4659-a65e-ae2c74629f33\" (UID: \"f5140852-5478-4659-a65e-ae2c74629f33\") " Nov 01 22:05:24.142365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.142341 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume" pod "f5140852-5478-4659-a65e-ae2c74629f33" (UID: "f5140852-5478-4659-a65e-ae2c74629f33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:24.142742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.142723 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5140852-5478-4659-a65e-ae2c74629f33-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:24.142821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.142748 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6z2x\" (UniqueName: \"kubernetes.io/projected/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb-kube-api-access-l6z2x\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:24.144305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.144283 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j" pod "f5140852-5478-4659-a65e-ae2c74629f33" (UID: "f5140852-5478-4659-a65e-ae2c74629f33"). InnerVolumeSpecName "kube-api-access-qhq6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:24.243936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.243842 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhq6j\" (UniqueName: \"kubernetes.io/projected/f5140852-5478-4659-a65e-ae2c74629f33-kube-api-access-qhq6j\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:24.928844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.928763 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" Nov 01 22:05:24.928844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.928772 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r" event={"ID":"f5140852-5478-4659-a65e-ae2c74629f33","Type":"ContainerDied","Data":"c57c2bd9c384229d5740ad42b78010ac1c1aed6c8556fc90f12d24e1778e68cc"} Nov 01 22:05:24.928844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.928838 4395 scope.go:122] "RemoveContainer" containerID="0a20f8e4abf95ad21b2883cd4d6b372231a202f150d91b9ff79ba1f490e28938" Nov 01 22:05:24.930452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.930427 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" Nov 01 22:05:24.930558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.930445 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd" event={"ID":"d0b4637b-d75b-4afd-be6e-92c3bdfcabbb","Type":"ContainerDied","Data":"93f61323a9eb5bcbd903d9db4612124c72c33ebedb5b1b9ef8d1aa9e9f8d1770"} Nov 01 22:05:24.940591 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.940566 4395 scope.go:122] "RemoveContainer" containerID="20ded8dca308fd97c90034ce2f93606eef83a0653460f902bd0d78482b79e75b" Nov 01 22:05:24.957779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.957756 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:05:24.962347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.962326 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-hs94r"] Nov 01 22:05:24.971997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.971971 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:05:24.976604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:24.976570 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-21dd16de-5b6fffb6c4-db9jd"] Nov 01 22:05:25.273805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:25.273773 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" path="/var/lib/kubelet/pods/d0b4637b-d75b-4afd-be6e-92c3bdfcabbb/volumes" Nov 01 22:05:25.274302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:25.274287 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5140852-5478-4659-a65e-ae2c74629f33" path="/var/lib/kubelet/pods/f5140852-5478-4659-a65e-ae2c74629f33/volumes" Nov 01 22:05:25.785789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:25.785746 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:05:28.453534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.453492 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:05:28.454443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454420 4395 cpu_manager.go:414] "containerMap: removing container" podUID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" containerName="payload-processing" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454444 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" containerName="payload-processing" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454461 4395 cpu_manager.go:414] "containerMap: removing container" podUID="422bea6e-d4ef-426a-9343-825cb0c93269" containerName="maas-api" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454468 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422bea6e-d4ef-426a-9343-825cb0c93269" containerName="maas-api" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454518 4395 cpu_manager.go:414] "containerMap: removing container" podUID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" containerName="payload-pre-processing" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454526 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" containerName="payload-pre-processing" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454550 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f5140852-5478-4659-a65e-ae2c74629f33" containerName="payload-pre-processing" Nov 01 22:05:28.454573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454557 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5140852-5478-4659-a65e-ae2c74629f33" containerName="payload-pre-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454578 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" containerName="payload-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454585 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" containerName="payload-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454796 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e5fb4-9f8d-44a7-926e-bf8995c60e52" containerName="payload-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454811 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="422bea6e-d4ef-426a-9343-825cb0c93269" containerName="maas-api" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454828 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="830982df-60fe-4f5b-a6a7-d2b1da6dc5a0" containerName="payload-pre-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454841 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5140852-5478-4659-a65e-ae2c74629f33" containerName="payload-pre-processing" Nov 01 22:05:28.454909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.454854 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b4637b-d75b-4afd-be6e-92c3bdfcabbb" containerName="payload-processing" Nov 01 22:05:28.460217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.460196 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.469010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.468988 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:05:28.488537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.488505 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bb882c-serving-cert\"" Nov 01 22:05:28.546857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.546820 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.547131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.547095 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.547193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.547168 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzp6x\" (UniqueName: \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.547428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.547401 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.662055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.659743 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.662055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.659940 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.662055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.661344 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.662055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.661413 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6x\" (UniqueName: \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.663113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.662715 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.705704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.705635 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.716177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.710299 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzp6x\" (UniqueName: \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.736282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.729936 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\") pod \"maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:28.784675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:28.781298 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:29.406224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:29.399375 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:05:30.093184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.093139 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" event={"ID":"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4","Type":"ContainerStarted","Data":"1ab433615970f440e64cdb3d30ce125e6eff56806a46d75bfc2cb4685715c02f"} Nov 01 22:05:30.093184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.093191 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" event={"ID":"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4","Type":"ContainerStarted","Data":"819f85f7747b04cdc188fde84d04a3e29becfac3d731c9f888a7deadc5ae2a0f"} Nov 01 22:05:30.093694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.093424 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:30.115820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.115763 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" podStartSLOduration=1.613568978 podStartE2EDuration="2.11575049s" totalImagesPullingTime="502.181512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:30.113143487 +0000 UTC m=+1277.087046043" watchObservedRunningTime="2026-11-01 22:05:30.11575049 +0000 UTC m=+1277.089653029" Nov 01 22:05:30.906907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.906868 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:05:30.911606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.911579 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:30.935796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.935767 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:05:30.983669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:30.983635 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-de0cba-openshift-default-dockercfg-rkxw4\"" Nov 01 22:05:31.103391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.103356 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.135290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.135256 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.145492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.145463 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.151051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.145804 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.151418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.151395 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmb9g\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.225167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.225077 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.225621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.225593 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.232129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.232100 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.232344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.232327 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.365054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.357915 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.365783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.365711 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374119 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374204 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374415 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374474 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374535 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.374723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374700 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.375178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.374796 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmb9g\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.375260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.362445 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.375898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.375872 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.376067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.375937 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.377630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.376424 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.377778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.369183 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.383873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.383830 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.393240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.393213 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmb9g\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.396360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.396318 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.396980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.396939 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\") pod \"e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.531674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.531645 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:31.871239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.871207 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:05:31.872946 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:05:31.872912 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835742f3_7833_4f1d_8bfd_8074f32a6b3c.slice/crio-64ca058d67c9089f88d474ff029f39c39fa99af20e0769b90f30bcfeca50c85d WatchSource:0}: Error finding container 64ca058d67c9089f88d474ff029f39c39fa99af20e0769b90f30bcfeca50c85d: Status 404 returned error can't find the container with id 64ca058d67c9089f88d474ff029f39c39fa99af20e0769b90f30bcfeca50c85d Nov 01 22:05:31.876266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.876231 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:31.876367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.876312 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:31.876367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:31.876360 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:32.185014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:32.184926 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerStarted","Data":"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc"} Nov 01 22:05:32.185014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:32.184991 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerStarted","Data":"64ca058d67c9089f88d474ff029f39c39fa99af20e0769b90f30bcfeca50c85d"} Nov 01 22:05:32.207573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:32.207516 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podStartSLOduration=2.207502539 podStartE2EDuration="2.207502539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:32.206437913 +0000 UTC m=+1279.180340453" watchObservedRunningTime="2026-11-01 22:05:32.207502539 +0000 UTC m=+1279.181405078" Nov 01 22:05:32.532208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:32.532166 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:32.537946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:32.537925 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:33.297981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.297950 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:33.304954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.304633 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:05:33.971336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.971067 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 22:05:33.972723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.971562 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" containerID="cri-o://3ce1029ea6d277b32e5a8805b141d2a2e85e20eb43accfae3ae3ecf7addf38c5" gracePeriod=30 Nov 01 22:05:33.976426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.976400 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 22:05:33.976657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:33.976623 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" containerID="cri-o://8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91" gracePeriod=30 Nov 01 22:05:35.842683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:35.842646 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:05:35.843086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:35.842700 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:05:36.308731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:36.283529 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:05:39.316715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.316687 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 22:05:39.333787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.333761 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.333957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.333795 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.333957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.333816 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334097 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.334234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334210 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334324 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334368 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.334405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334371 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334629 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334687 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.334758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.334721 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\") pod \"40362c5d-e7d8-408e-8961-71efe6d7f16f\" (UID: \"40362c5d-e7d8-408e-8961-71efe6d7f16f\") " Nov 01 22:05:39.335711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.335525 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.335711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.335548 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.335711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.335553 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:39.335711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.335678 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.336013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.335992 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.337345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.337324 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "kube-api-access-xd57f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:39.337553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.337524 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:39.337553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.337545 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:39.337714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.337690 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy" pod "40362c5d-e7d8-408e-8961-71efe6d7f16f" (UID: "40362c5d-e7d8-408e-8961-71efe6d7f16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.436846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436810 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.436846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436842 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.436846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436854 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd57f\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-kube-api-access-xd57f\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.437153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436863 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.437153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436875 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40362c5d-e7d8-408e-8961-71efe6d7f16f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.437153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436884 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.437153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.436892 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40362c5d-e7d8-408e-8961-71efe6d7f16f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.643224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.643192 4395 generic.go:356] "Generic (PLEG): container finished" podID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerID="8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91" exitCode=0 Nov 01 22:05:39.643454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.643416 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" event={"ID":"40362c5d-e7d8-408e-8961-71efe6d7f16f","Type":"ContainerDied","Data":"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91"} Nov 01 22:05:39.643564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.643494 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" event={"ID":"40362c5d-e7d8-408e-8961-71efe6d7f16f","Type":"ContainerDied","Data":"e03fd62a95e49e1940be0079a6b2e7b7aab6948268f59fb6caf46bc485bb29d7"} Nov 01 22:05:39.643564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.643531 4395 scope.go:122] "RemoveContainer" containerID="8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91" Nov 01 22:05:39.643758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.643741 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq" Nov 01 22:05:39.670260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.656400 4395 generic.go:356] "Generic (PLEG): container finished" podID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerID="3ce1029ea6d277b32e5a8805b141d2a2e85e20eb43accfae3ae3ecf7addf38c5" exitCode=0 Nov 01 22:05:39.670260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.656628 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" event={"ID":"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf","Type":"ContainerDied","Data":"3ce1029ea6d277b32e5a8805b141d2a2e85e20eb43accfae3ae3ecf7addf38c5"} Nov 01 22:05:39.694088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.693508 4395 scope.go:122] "RemoveContainer" containerID="8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91" Nov 01 22:05:39.694088 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:39.693904 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91\": container with ID starting with 8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91 not found: ID does not exist" containerID="8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91" Nov 01 22:05:39.694088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.693943 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91"} err="failed to get container status \"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91\": rpc error: code = NotFound desc = could not find container \"8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91\": container with ID starting with 8e4cf9087e5e00e869a5286e18991d34ee78a8f34506328b7a2312dc9a8f5c91 not found: ID does not exist" Nov 01 22:05:39.700725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.700701 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 22:05:39.705283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.705261 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-j9vlq"] Nov 01 22:05:39.754003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.753985 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 22:05:39.843865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.843820 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.843926 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.843999 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844064 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844159 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844203 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844262 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.844398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844310 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.844398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844450 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844517 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\") pod \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\" (UID: \"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf\") " Nov 01 22:05:39.844664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844598 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:39.844664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844634 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.844961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.844937 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.845316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.845294 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.845420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.845321 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.845420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.845337 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.845420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.845350 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.845420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.845363 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.847837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.847811 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:05:39.847837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.847810 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:05:39.847961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.847905 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "kube-api-access-sgvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:39.848007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.847970 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token" pod "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" (UID: "3e474bb8-1d3e-4bab-8514-1931f8bcc3cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:39.946874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.946800 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvtw\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-kube-api-access-sgvtw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.946874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.946827 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.946874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.946837 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:39.946874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:39.946847 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:40.668750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:40.668723 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" Nov 01 22:05:40.669323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:40.668814 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97" event={"ID":"3e474bb8-1d3e-4bab-8514-1931f8bcc3cf","Type":"ContainerDied","Data":"0c2ffefc676dbacf5a2dfe54b44b65d5651a7c0d1d273f208041398ef69064ee"} Nov 01 22:05:40.669323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:40.668888 4395 scope.go:122] "RemoveContainer" containerID="3ce1029ea6d277b32e5a8805b141d2a2e85e20eb43accfae3ae3ecf7addf38c5" Nov 01 22:05:40.717074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:40.717042 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 22:05:40.720762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:40.720736 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-zvw97"] Nov 01 22:05:41.166938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:41.166905 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" path="/var/lib/kubelet/pods/3e474bb8-1d3e-4bab-8514-1931f8bcc3cf/volumes" Nov 01 22:05:41.167513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:41.167496 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" path="/var/lib/kubelet/pods/40362c5d-e7d8-408e-8961-71efe6d7f16f/volumes" Nov 01 22:05:45.742330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:45.742294 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:05:45.742995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:45.742592 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" podUID="3ed79f4b-8fbd-4018-8516-80e127799b53" containerName="maas-api" containerID="cri-o://7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5" gracePeriod=30 Nov 01 22:05:46.344256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.344229 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:05:46.387431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.387400 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\") pod \"3ed79f4b-8fbd-4018-8516-80e127799b53\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " Nov 01 22:05:46.387648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.387634 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\") pod \"3ed79f4b-8fbd-4018-8516-80e127799b53\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " Nov 01 22:05:46.387742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.387685 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\") pod \"3ed79f4b-8fbd-4018-8516-80e127799b53\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " Nov 01 22:05:46.387742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.387706 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\" (UniqueName: \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\") pod \"3ed79f4b-8fbd-4018-8516-80e127799b53\" (UID: \"3ed79f4b-8fbd-4018-8516-80e127799b53\") " Nov 01 22:05:46.388153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.388122 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca" pod "3ed79f4b-8fbd-4018-8516-80e127799b53" (UID: "3ed79f4b-8fbd-4018-8516-80e127799b53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:05:46.388635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.388507 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed79f4b-8fbd-4018-8516-80e127799b53-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:46.392076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.390213 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs" pod "3ed79f4b-8fbd-4018-8516-80e127799b53" (UID: "3ed79f4b-8fbd-4018-8516-80e127799b53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:46.392076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.390370 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls" pod "3ed79f4b-8fbd-4018-8516-80e127799b53" (UID: "3ed79f4b-8fbd-4018-8516-80e127799b53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:05:46.392076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.390510 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg" pod "3ed79f4b-8fbd-4018-8516-80e127799b53" (UID: "3ed79f4b-8fbd-4018-8516-80e127799b53"). InnerVolumeSpecName "kube-api-access-ltdwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:05:46.528735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.517504 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:46.528735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.517628 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdwg\" (UniqueName: \"kubernetes.io/projected/3ed79f4b-8fbd-4018-8516-80e127799b53-kube-api-access-ltdwg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:46.528735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.517651 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed79f4b-8fbd-4018-8516-80e127799b53-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:05:46.763716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.763682 4395 generic.go:356] "Generic (PLEG): container finished" podID="3ed79f4b-8fbd-4018-8516-80e127799b53" containerID="7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5" exitCode=0 Nov 01 22:05:46.764281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.763727 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" Nov 01 22:05:46.764281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.763763 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" event={"ID":"3ed79f4b-8fbd-4018-8516-80e127799b53","Type":"ContainerDied","Data":"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5"} Nov 01 22:05:46.764281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.763825 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz" event={"ID":"3ed79f4b-8fbd-4018-8516-80e127799b53","Type":"ContainerDied","Data":"489b943c75573aa46018b6dbbd1c413080c9c83841f80303b372a10a45f9e3d7"} Nov 01 22:05:46.764281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.763849 4395 scope.go:122] "RemoveContainer" containerID="7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5" Nov 01 22:05:46.779443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.779424 4395 scope.go:122] "RemoveContainer" containerID="7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5" Nov 01 22:05:46.779732 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:46.779713 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5\": container with ID starting with 7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5 not found: ID does not exist" containerID="7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5" Nov 01 22:05:46.779780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.779741 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5"} err="failed to get container status \"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5\": rpc error: code = NotFound desc = could not find container \"7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5\": container with ID starting with 7e9f0c91b4b0b86f69ad02ac0f894c7ab7ac3d371e95072fafc99f513dba11f5 not found: ID does not exist" Nov 01 22:05:46.793951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.793929 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:05:46.799886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:46.799862 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d506c6-698645c5b9-8fjqz"] Nov 01 22:05:47.166813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:47.166731 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed79f4b-8fbd-4018-8516-80e127799b53" path="/var/lib/kubelet/pods/3ed79f4b-8fbd-4018-8516-80e127799b53/volumes" Nov 01 22:05:49.778445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.778413 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:05:49.779459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779440 4395 cpu_manager.go:414] "containerMap: removing container" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" Nov 01 22:05:49.779535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779460 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" Nov 01 22:05:49.779535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779479 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3ed79f4b-8fbd-4018-8516-80e127799b53" containerName="maas-api" Nov 01 22:05:49.779535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779484 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed79f4b-8fbd-4018-8516-80e127799b53" containerName="maas-api" Nov 01 22:05:49.779535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779508 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" Nov 01 22:05:49.779535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779513 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" Nov 01 22:05:49.779696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779610 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed79f4b-8fbd-4018-8516-80e127799b53" containerName="maas-api" Nov 01 22:05:49.779696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779622 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e474bb8-1d3e-4bab-8514-1931f8bcc3cf" containerName="istio-proxy" Nov 01 22:05:49.779696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.779633 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="40362c5d-e7d8-408e-8961-71efe6d7f16f" containerName="istio-proxy" Nov 01 22:05:49.784965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.784946 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:49.794383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.794354 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:05:49.796105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.796078 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-de0cba-serving-cert\"" Nov 01 22:05:49.922968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.922940 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:49.923166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.922977 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2cc\" (UniqueName: \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:49.923166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.923099 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:49.923248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:49.923201 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.028060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.025499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.028060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.025551 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2cc\" (UniqueName: \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.028060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.025640 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.028060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.025674 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.028060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.026651 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.032087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.030268 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.059963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.059883 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.063726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.063666 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2cc\" (UniqueName: \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\") pod \"maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.103311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.103284 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:50.407716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.407609 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:05:50.428435 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:05:50.428406 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cba018a_1697_41fd_9acd_46ce936433c3.slice/crio-a24d9017c24c86a32cb7d88abbaba901f05be95d77d10a526721972b7798a57b WatchSource:0}: Error finding container a24d9017c24c86a32cb7d88abbaba901f05be95d77d10a526721972b7798a57b: Status 404 returned error can't find the container with id a24d9017c24c86a32cb7d88abbaba901f05be95d77d10a526721972b7798a57b Nov 01 22:05:50.814521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:50.814478 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" event={"ID":"7cba018a-1697-41fd-9acd-46ce936433c3","Type":"ContainerStarted","Data":"a24d9017c24c86a32cb7d88abbaba901f05be95d77d10a526721972b7798a57b"} Nov 01 22:05:51.824538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:51.824493 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" event={"ID":"7cba018a-1697-41fd-9acd-46ce936433c3","Type":"ContainerStarted","Data":"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0"} Nov 01 22:05:51.825081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:51.824697 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:51.847017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:51.846950 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" podStartSLOduration=2.359610602 podStartE2EDuration="2.846928858s" totalImagesPullingTime="487.318256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:51.844339849 +0000 UTC m=+1298.818242407" watchObservedRunningTime="2026-11-01 22:05:51.846928858 +0000 UTC m=+1298.820831398" Nov 01 22:05:52.303058 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:05:52.302984 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:05:53.755329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.755295 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:05:53.759986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.759967 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:53.771424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.770987 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:05:53.776900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.776873 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18657f-dockercfg-l22m7\"" Nov 01 22:05:53.823381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.823352 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18657f\"" Nov 01 22:05:53.954052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.953123 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:53.954052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:53.953201 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk4n\" (UniqueName: \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.058388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.058293 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.058648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.058627 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk4n\" (UniqueName: \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.076068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.060279 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.088057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.080156 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk4n\" (UniqueName: \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\") pod \"payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.380840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.380713 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:54.727360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.714231 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:05:54.848701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:54.848673 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" event={"ID":"7330bb1f-371f-478e-af31-3e83f7ec5704","Type":"ContainerStarted","Data":"e08fb80aa35c9477db38e13ffd616c0cb11c9c01ee592e920b2ac2ed296e7011"} Nov 01 22:05:55.490056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.489002 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:05:55.515051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.506603 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.515051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.508885 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:05:55.550905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.550879 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\"" Nov 01 22:05:55.701072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701002 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701185 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701231 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701379 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701427 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701548 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701611 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8sjb\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701675 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.701744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.701732 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:55.821777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.821735 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.127805 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8sjb\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.127889 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.127938 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128157 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128284 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128335 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128404 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128448 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.128883 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.132138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:55.824096 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.136526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.136390 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.136684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.136639 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.137555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.137510 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.142005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.141976 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.146128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.146096 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.148168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.148142 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.150771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.150739 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" event={"ID":"7330bb1f-371f-478e-af31-3e83f7ec5704","Type":"ContainerStarted","Data":"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611"} Nov 01 22:05:56.151415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.151396 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:05:56.153193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.153166 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8sjb\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.190685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.190626 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" podStartSLOduration=3.190608016 podStartE2EDuration="3.190608016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:56.184732347 +0000 UTC m=+1303.158634886" watchObservedRunningTime="2026-11-01 22:05:56.190608016 +0000 UTC m=+1303.164510558" Nov 01 22:05:56.432494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.432410 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:56.744941 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:05:56.744891 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a39204_5e0e_4a1e_a778_042361d17445.slice/crio-e5dcd3c5edeee63959b942f3449e0e19036b632ec6b52ec0c8f987c0f0d082dc WatchSource:0}: Error finding container e5dcd3c5edeee63959b942f3449e0e19036b632ec6b52ec0c8f987c0f0d082dc: Status 404 returned error can't find the container with id e5dcd3c5edeee63959b942f3449e0e19036b632ec6b52ec0c8f987c0f0d082dc Nov 01 22:05:56.745537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.745511 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:05:56.748416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.748312 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:56.748416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.748403 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:56.748593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:56.748449 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:05:57.167487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.167445 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" event={"ID":"01a39204-5e0e-4a1e-a778-042361d17445","Type":"ContainerStarted","Data":"ab909b1759b4959049283377c14d8a2ca941a63ba69b9e09ce9fa8ae3631ad07"} Nov 01 22:05:57.167877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.167506 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" event={"ID":"01a39204-5e0e-4a1e-a778-042361d17445","Type":"ContainerStarted","Data":"e5dcd3c5edeee63959b942f3449e0e19036b632ec6b52ec0c8f987c0f0d082dc"} Nov 01 22:05:57.185211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.184906 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podStartSLOduration=2.184888314 podStartE2EDuration="2.184888314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:57.184731679 +0000 UTC m=+1304.158634229" watchObservedRunningTime="2026-11-01 22:05:57.184888314 +0000 UTC m=+1304.158790855" Nov 01 22:05:57.432996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.432918 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:05:57.434661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.434632 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:05:57.434801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.434691 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:05:57.845765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:57.842773 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:05:58.433353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:58.433321 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:05:58.433732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:58.433369 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:05:59.449064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:59.445174 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:05:59.449064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:05:59.445239 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:06:00.452179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:00.446433 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:06:00.452179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:00.446493 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:06:01.433751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:01.433714 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:06:01.433925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:01.433775 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:06:02.433604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:02.433548 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:06:02.434136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:02.433632 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:06:03.433623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:03.433584 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 01 22:06:03.434225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:03.433647 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 01 22:06:04.438320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:04.438291 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:06:04.438806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:04.438782 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:06:04.439391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:04.439371 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:06:07.167213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:07.167180 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:06:08.417948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:08.417914 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:06:08.418485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:08.418179 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" containerID="cri-o://5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234" gracePeriod=30 Nov 01 22:06:10.481655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.481614 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:10.540783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.540698 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.551252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.551222 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:10.636619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636584 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.636777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636626 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.636840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636770 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.636921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636900 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.636979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636935 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.636979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.636950 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.637111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.637091 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.637166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.637147 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.637205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.637178 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4872\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738064 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738112 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738149 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4872\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738288 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738327 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738409 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738522 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738457 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738522 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738547 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738580 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738605 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738680 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.738862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.738829 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.739142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.739125 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.740807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.740782 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.741069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.741052 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.750235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.750208 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4872\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.750404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.750326 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:10.883058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:10.881343 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:11.355849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.355820 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:11.357614 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:06:11.357585 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e74a22_77dd_4ad1_aca8_e27ec645426e.slice/crio-f7fbbb981723cc65483545d9b634c8da161bad64fff850768cc53690ff10074d WatchSource:0}: Error finding container f7fbbb981723cc65483545d9b634c8da161bad64fff850768cc53690ff10074d: Status 404 returned error can't find the container with id f7fbbb981723cc65483545d9b634c8da161bad64fff850768cc53690ff10074d Nov 01 22:06:11.360729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.360696 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:11.360817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.360764 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:11.360817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.360794 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:11.480821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.480778 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" event={"ID":"58e74a22-77dd-4ad1-aca8-e27ec645426e","Type":"ContainerStarted","Data":"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c"} Nov 01 22:06:11.481079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.480855 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" event={"ID":"58e74a22-77dd-4ad1-aca8-e27ec645426e","Type":"ContainerStarted","Data":"f7fbbb981723cc65483545d9b634c8da161bad64fff850768cc53690ff10074d"} Nov 01 22:06:11.516830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.509175 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" podStartSLOduration=1.509154824 podStartE2EDuration="1.509154824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:11.503321599 +0000 UTC m=+1318.477224151" watchObservedRunningTime="2026-11-01 22:06:11.509154824 +0000 UTC m=+1318.483057365" Nov 01 22:06:11.881994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:11.881907 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:12.904050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:12.902170 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 22:06:12.904050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:12.902224 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 22:06:13.360596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.360566 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:06:13.361090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.361059 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" containerID="cri-o://ab909b1759b4959049283377c14d8a2ca941a63ba69b9e09ce9fa8ae3631ad07" gracePeriod=30 Nov 01 22:06:13.362718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.362692 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:13.588606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.584430 4395 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" secret="" err="secret \"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\" not found" Nov 01 22:06:13.882712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.882676 4395 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 01 22:06:13.883407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:13.882977 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 01 22:06:14.055068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.054909 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:06:14.130571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130486 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130555 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130604 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130652 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130690 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130739 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130769 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.130859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130818 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:14.131207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.130870 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.131207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.131062 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:14.131315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.131221 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:14.131315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.131235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:14.131315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.131293 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\") pod \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\" (UID: \"8ea0c9e9-ce17-413a-9efa-feaaf0f57802\") " Nov 01 22:06:14.131969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.131941 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:14.132262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.132238 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.132343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.132269 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.132343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.132284 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.132343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.132298 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.132343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.132314 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.135657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.135628 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:14.135779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.135727 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:14.135840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.135805 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:14.136308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.136274 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x" pod "8ea0c9e9-ce17-413a-9efa-feaaf0f57802" (UID: "8ea0c9e9-ce17-413a-9efa-feaaf0f57802"). InnerVolumeSpecName "kube-api-access-j4l5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:14.238416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.236095 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4l5x\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-kube-api-access-j4l5x\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.238416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.236126 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.238416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.236142 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.238416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.236157 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea0c9e9-ce17-413a-9efa-feaaf0f57802-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:14.613110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.607412 4395 generic.go:356] "Generic (PLEG): container finished" podID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerID="5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234" exitCode=0 Nov 01 22:06:14.613110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.607724 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" containerID="cri-o://b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c" gracePeriod=30 Nov 01 22:06:14.613110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.608206 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" Nov 01 22:06:14.613110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.610156 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerDied","Data":"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234"} Nov 01 22:06:14.631158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.624199 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r" event={"ID":"8ea0c9e9-ce17-413a-9efa-feaaf0f57802","Type":"ContainerDied","Data":"a7d63cf893f73a238fb25235b688d69a40940f45e1f8fb5a5289bacd3dd1c1aa"} Nov 01 22:06:14.631158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.624273 4395 scope.go:122] "RemoveContainer" containerID="5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234" Nov 01 22:06:14.699298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.695745 4395 scope.go:122] "RemoveContainer" containerID="4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c" Nov 01 22:06:14.756337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.756313 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:06:14.758893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.758878 4395 scope.go:122] "RemoveContainer" containerID="5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234" Nov 01 22:06:14.759431 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:14.759405 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234\": container with ID starting with 5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234 not found: ID does not exist" containerID="5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234" Nov 01 22:06:14.759552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.759469 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234"} err="failed to get container status \"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234\": rpc error: code = NotFound desc = could not find container \"5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234\": container with ID starting with 5db0c9435fb2d11b563005d468e4259157bae8e98fe10321105086baae664234 not found: ID does not exist" Nov 01 22:06:14.759552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.759498 4395 scope.go:122] "RemoveContainer" containerID="4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c" Nov 01 22:06:14.759863 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:14.759847 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c\": container with ID starting with 4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c not found: ID does not exist" containerID="4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c" Nov 01 22:06:14.759959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.759942 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c"} err="failed to get container status \"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c\": rpc error: code = NotFound desc = could not find container \"4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c\": container with ID starting with 4a1baf3c15e4ee53e0379b5b6666b43fc52a8f45843e1856767bebace3b8072c not found: ID does not exist" Nov 01 22:06:14.761075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:14.761051 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a19582-openshift-default-7df57db669-fzn8r"] Nov 01 22:06:15.167357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:15.167279 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" path="/var/lib/kubelet/pods/8ea0c9e9-ce17-413a-9efa-feaaf0f57802/volumes" Nov 01 22:06:18.717538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.717504 4395 generic.go:356] "Generic (PLEG): container finished" podID="01a39204-5e0e-4a1e-a778-042361d17445" containerID="ab909b1759b4959049283377c14d8a2ca941a63ba69b9e09ce9fa8ae3631ad07" exitCode=0 Nov 01 22:06:18.717971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.717535 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" event={"ID":"01a39204-5e0e-4a1e-a778-042361d17445","Type":"ContainerDied","Data":"ab909b1759b4959049283377c14d8a2ca941a63ba69b9e09ce9fa8ae3631ad07"} Nov 01 22:06:18.822605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.822577 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:06:18.857742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.857710 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.863080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.858263 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:18.863400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.863364 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.863735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.863707 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:18.863834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.863740 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.863834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.863786 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.863944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.863873 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.864077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864052 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:18.864155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864107 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.864225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864207 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.864305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864289 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.864433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864402 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\") pod \"01a39204-5e0e-4a1e-a778-042361d17445\" (UID: \"01a39204-5e0e-4a1e-a778-042361d17445\") " Nov 01 22:06:18.864570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864520 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:18.864570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.864539 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:18.865341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.865316 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.865442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.865347 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a39204-5e0e-4a1e-a778-042361d17445-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.865442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.865363 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.865442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.865377 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.865442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.865400 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.867477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.867443 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:18.869072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.868476 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:18.869072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.868583 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "kube-api-access-r8sjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:18.870476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.870455 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token" pod "01a39204-5e0e-4a1e-a778-042361d17445" (UID: "01a39204-5e0e-4a1e-a778-042361d17445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:18.967065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.966963 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.967065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.966990 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8sjb\" (UniqueName: \"kubernetes.io/projected/01a39204-5e0e-4a1e-a778-042361d17445-kube-api-access-r8sjb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.967065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.967000 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a39204-5e0e-4a1e-a778-042361d17445-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:18.967065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:18.967008 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a39204-5e0e-4a1e-a778-042361d17445-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:19.518367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.518334 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:06:19.538432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.538400 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:06:19.726808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.726761 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" event={"ID":"01a39204-5e0e-4a1e-a778-042361d17445","Type":"ContainerDied","Data":"e5dcd3c5edeee63959b942f3449e0e19036b632ec6b52ec0c8f987c0f0d082dc"} Nov 01 22:06:19.727300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.726813 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr" Nov 01 22:06:19.727300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.726837 4395 scope.go:122] "RemoveContainer" containerID="ab909b1759b4959049283377c14d8a2ca941a63ba69b9e09ce9fa8ae3631ad07" Nov 01 22:06:19.749589 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.749560 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:06:19.754520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:19.754486 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-gqqjr"] Nov 01 22:06:21.167257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:21.167227 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a39204-5e0e-4a1e-a778-042361d17445" path="/var/lib/kubelet/pods/01a39204-5e0e-4a1e-a778-042361d17445/volumes" Nov 01 22:06:22.376701 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:22.376660 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:06:24.406869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.400765 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:24.509807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.509774 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.509995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.509844 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.509995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.509905 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.509995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.509942 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.509995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.509971 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.510293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510047 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:24.510293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510074 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.510293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510176 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.510293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510252 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.510507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510314 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\") pod \"58e74a22-77dd-4ad1-aca8-e27ec645426e\" (UID: \"58e74a22-77dd-4ad1-aca8-e27ec645426e\") " Nov 01 22:06:24.510507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510370 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:24.510507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510448 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:24.510507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510460 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:24.510907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510587 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:24.511000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.510975 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e74a22-77dd-4ad1-aca8-e27ec645426e-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.511099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.511008 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.511099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.511070 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.511099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.511086 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.511266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.511099 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.512445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.512416 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:24.512553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.512453 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "kube-api-access-q4872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:24.512553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.512487 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:24.512748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.512730 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token" pod "58e74a22-77dd-4ad1-aca8-e27ec645426e" (UID: "58e74a22-77dd-4ad1-aca8-e27ec645426e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:24.518857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.518813 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" podUID="5c2d293b-459f-4f60-9964-c967b95a51e8" containerName="payload-processing" containerID="cri-o://a4401a2a7ecfcf1d5b05e63b385b5ccc5e51c8a0188818bac7d4b210d8c53d1b" gracePeriod=25 Nov 01 22:06:24.538980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.538951 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" podUID="2207a59f-da22-43f1-9b51-4d040376d962" containerName="payload-pre-processing" containerID="cri-o://9fd6a6d80dc48090970ec516084d340b2c3d6d330b14a34f79ec911530a4efcb" gracePeriod=25 Nov 01 22:06:24.611983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.611960 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.612097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.611986 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.612097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.611996 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e74a22-77dd-4ad1-aca8-e27ec645426e-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.612097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.612005 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4872\" (UniqueName: \"kubernetes.io/projected/58e74a22-77dd-4ad1-aca8-e27ec645426e-kube-api-access-q4872\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.769928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.769889 4395 generic.go:356] "Generic (PLEG): container finished" podID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerID="b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c" exitCode=0 Nov 01 22:06:24.770409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.770161 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" event={"ID":"58e74a22-77dd-4ad1-aca8-e27ec645426e","Type":"ContainerDied","Data":"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c"} Nov 01 22:06:24.770650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.770627 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" event={"ID":"58e74a22-77dd-4ad1-aca8-e27ec645426e","Type":"ContainerDied","Data":"f7fbbb981723cc65483545d9b634c8da161bad64fff850768cc53690ff10074d"} Nov 01 22:06:24.773197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.773175 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" event={"ID":"2207a59f-da22-43f1-9b51-4d040376d962","Type":"ContainerDied","Data":"9fd6a6d80dc48090970ec516084d340b2c3d6d330b14a34f79ec911530a4efcb"} Nov 01 22:06:24.773349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.770243 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr" Nov 01 22:06:24.774695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.773054 4395 generic.go:356] "Generic (PLEG): container finished" podID="2207a59f-da22-43f1-9b51-4d040376d962" containerID="9fd6a6d80dc48090970ec516084d340b2c3d6d330b14a34f79ec911530a4efcb" exitCode=0 Nov 01 22:06:24.774695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.770908 4395 scope.go:122] "RemoveContainer" containerID="b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c" Nov 01 22:06:24.780061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.780001 4395 generic.go:356] "Generic (PLEG): container finished" podID="5c2d293b-459f-4f60-9964-c967b95a51e8" containerID="a4401a2a7ecfcf1d5b05e63b385b5ccc5e51c8a0188818bac7d4b210d8c53d1b" exitCode=0 Nov 01 22:06:24.780272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.780102 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" event={"ID":"5c2d293b-459f-4f60-9964-c967b95a51e8","Type":"ContainerDied","Data":"a4401a2a7ecfcf1d5b05e63b385b5ccc5e51c8a0188818bac7d4b210d8c53d1b"} Nov 01 22:06:24.790657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.790587 4395 scope.go:122] "RemoveContainer" containerID="b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c" Nov 01 22:06:24.791362 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:24.791335 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c\": container with ID starting with b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c not found: ID does not exist" containerID="b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c" Nov 01 22:06:24.791486 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.791371 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c"} err="failed to get container status \"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c\": rpc error: code = NotFound desc = could not find container \"b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c\": container with ID starting with b12a5706bc2e06827359a78f688d16082de45890a252efcdd876a29ed37d1d6c not found: ID does not exist" Nov 01 22:06:24.809176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.809111 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:06:24.820274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.820249 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:24.825716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.825692 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xdrkr"] Nov 01 22:06:24.888485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.888451 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:06:24.924144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.924078 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\") pod \"5c2d293b-459f-4f60-9964-c967b95a51e8\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " Nov 01 22:06:24.924297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.924177 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\" (UniqueName: \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\") pod \"5c2d293b-459f-4f60-9964-c967b95a51e8\" (UID: \"5c2d293b-459f-4f60-9964-c967b95a51e8\") " Nov 01 22:06:24.924487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.924463 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume" pod "5c2d293b-459f-4f60-9964-c967b95a51e8" (UID: "5c2d293b-459f-4f60-9964-c967b95a51e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:24.924948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.924869 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d293b-459f-4f60-9964-c967b95a51e8-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:24.926526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:24.926494 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw" pod "5c2d293b-459f-4f60-9964-c967b95a51e8" (UID: "5c2d293b-459f-4f60-9964-c967b95a51e8"). InnerVolumeSpecName "kube-api-access-9msfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:25.025399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.025372 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\") pod \"2207a59f-da22-43f1-9b51-4d040376d962\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " Nov 01 22:06:25.025573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.025414 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\" (UniqueName: \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\") pod \"2207a59f-da22-43f1-9b51-4d040376d962\" (UID: \"2207a59f-da22-43f1-9b51-4d040376d962\") " Nov 01 22:06:25.025743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.025723 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume" pod "2207a59f-da22-43f1-9b51-4d040376d962" (UID: "2207a59f-da22-43f1-9b51-4d040376d962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:25.026018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.026001 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2207a59f-da22-43f1-9b51-4d040376d962-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:25.026132 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.026038 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msfw\" (UniqueName: \"kubernetes.io/projected/5c2d293b-459f-4f60-9964-c967b95a51e8-kube-api-access-9msfw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:25.027405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.027383 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r" pod "2207a59f-da22-43f1-9b51-4d040376d962" (UID: "2207a59f-da22-43f1-9b51-4d040376d962"). InnerVolumeSpecName "kube-api-access-zfr5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:25.145450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.128547 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfr5r\" (UniqueName: \"kubernetes.io/projected/2207a59f-da22-43f1-9b51-4d040376d962-kube-api-access-zfr5r\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:25.208288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.207178 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" path="/var/lib/kubelet/pods/58e74a22-77dd-4ad1-aca8-e27ec645426e/volumes" Nov 01 22:06:25.868052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.867790 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" event={"ID":"2207a59f-da22-43f1-9b51-4d040376d962","Type":"ContainerDied","Data":"ab6ec5703a1a6563846a016c2c09f6c09434ef0d7b4a7618cc46b06528ae8d20"} Nov 01 22:06:25.868052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.867864 4395 scope.go:122] "RemoveContainer" containerID="9fd6a6d80dc48090970ec516084d340b2c3d6d330b14a34f79ec911530a4efcb" Nov 01 22:06:25.868052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.867986 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts" Nov 01 22:06:25.892512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.890400 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" event={"ID":"5c2d293b-459f-4f60-9964-c967b95a51e8","Type":"ContainerDied","Data":"2b7293bc1787ce95a51858e055ac43d4f0d4f91563e64ebc1532874b4c3ab467"} Nov 01 22:06:25.892512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.890517 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr" Nov 01 22:06:25.937397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.937355 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:06:25.995061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:25.982072 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0390e1-6b87bf79c5-4plts"] Nov 01 22:06:26.001202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:26.001017 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:06:26.001288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:26.001217 4395 scope.go:122] "RemoveContainer" containerID="a4401a2a7ecfcf1d5b05e63b385b5ccc5e51c8a0188818bac7d4b210d8c53d1b" Nov 01 22:06:26.018442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:26.018224 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0390e1-7b5b74b5c6-dvxpr"] Nov 01 22:06:27.353050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:27.345702 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2207a59f-da22-43f1-9b51-4d040376d962" path="/var/lib/kubelet/pods/2207a59f-da22-43f1-9b51-4d040376d962/volumes" Nov 01 22:06:27.353050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:27.346596 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2d293b-459f-4f60-9964-c967b95a51e8" path="/var/lib/kubelet/pods/5c2d293b-459f-4f60-9964-c967b95a51e8/volumes" Nov 01 22:06:28.087052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:28.072148 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:28.139169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:28.139127 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:06:28.232207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:28.232162 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:28.256053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:28.253306 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.002160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.002118 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.073172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.073113 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.074109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.074087 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:06:29.311113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.291933 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:06:29.311814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.295005 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:06:29.384729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.384688 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.385517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.385487 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.585118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.583191 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:29.795994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.793797 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.227142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:29.967799 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:06:30.272860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.270294 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:06:30.272860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.270403 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:06:30.272860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.270451 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:06:30.315059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.153200 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.315059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.049666 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.315059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.294620 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerName="maas-api" containerID="cri-o://9c3982c2eec3ad606e25671e0de3affe253fede0d326e02dc5d98c0d3dbede27" gracePeriod=30 Nov 01 22:06:30.315059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.218129 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.315059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.309782 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:06:30.398051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.397453 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.416318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.398666 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:06:30.433187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.433123 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:30.540080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.514176 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:06:30.540080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.514450 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:06:30.540080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.515074 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:06:30.592295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.592254 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 01 22:06:30.669792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.665438 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.116:8443/health\": dial tcp 10.133.0.116:8443: connect: connection refused" Nov 01 22:06:30.829109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.795107 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:06:30.946053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:30.938906 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:06:31.116823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.115641 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.323054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.292276 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 01 22:06:31.405118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.313814 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.405118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.398767 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.405118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.399805 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.521805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.521763 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.616019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.531153 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:31.616019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.582692 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:06:31.616019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.583855 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:06:31.809703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.790834 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 22:06:31.830691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.829957 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:06:31.830691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.830022 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:06:31.830691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.830077 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:06:31.915313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.906660 4395 generic.go:356] "Generic (PLEG): container finished" podID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerID="9c3982c2eec3ad606e25671e0de3affe253fede0d326e02dc5d98c0d3dbede27" exitCode=0 Nov 01 22:06:31.948234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.916661 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" event={"ID":"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410","Type":"ContainerDied","Data":"9c3982c2eec3ad606e25671e0de3affe253fede0d326e02dc5d98c0d3dbede27"} Nov 01 22:06:31.971694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.971647 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:06:31.971971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.971953 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:06:31.972147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:31.942933 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:32.035678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.026602 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:06:32.037012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.036991 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:06:32.040358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.039653 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:06:32.040358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.040058 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:06:32.046503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.046478 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:06:32.165597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.165469 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\") pod \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " Nov 01 22:06:32.165755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.165657 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\") pod \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " Nov 01 22:06:32.165755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.165731 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") pod \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " Nov 01 22:06:32.165905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.165886 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\" (UniqueName: \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\") pod \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\" (UID: \"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410\") " Nov 01 22:06:32.165957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.165913 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca" pod "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" (UID: "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:32.166889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.166859 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:32.168347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.168320 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs" pod "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" (UID: "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:32.168347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.168320 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw" pod "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" (UID: "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410"). InnerVolumeSpecName "kube-api-access-kcnpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:32.168470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.168442 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls" pod "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" (UID: "2d9afd5e-0e61-4b1b-9326-d1c86e4a8410"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:06:32.269468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.269433 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:32.269678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.269661 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:32.269777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:32.269763 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnpw\" (UniqueName: \"kubernetes.io/projected/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410-kube-api-access-kcnpw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:33.048399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.045888 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" Nov 01 22:06:33.048399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.045900 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf" event={"ID":"2d9afd5e-0e61-4b1b-9326-d1c86e4a8410","Type":"ContainerDied","Data":"9def4b90fbfdd54b9306e758e529af2d187927744e541492bc0cb306764691cf"} Nov 01 22:06:33.048399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.045957 4395 scope.go:122] "RemoveContainer" containerID="9c3982c2eec3ad606e25671e0de3affe253fede0d326e02dc5d98c0d3dbede27" Nov 01 22:06:33.139137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.128690 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:06:33.156590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.156557 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-004180-76cfcdc75c-6ntbf"] Nov 01 22:06:33.240886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.240844 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" path="/var/lib/kubelet/pods/2d9afd5e-0e61-4b1b-9326-d1c86e4a8410/volumes" Nov 01 22:06:33.908512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.908466 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 22:06:33.908736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:33.908705 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" containerID="cri-o://b6df81d7b852d637b8f2fc2ebd5439df41fc95cd8beb3b3b30f1f15998741023" gracePeriod=30 Nov 01 22:06:34.770303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.770271 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:06:34.771246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771225 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:34.771246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771247 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771271 4395 cpu_manager.go:414] "containerMap: removing container" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771278 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771289 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerName="maas-api" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771296 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerName="maas-api" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771310 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5c2d293b-459f-4f60-9964-c967b95a51e8" containerName="payload-processing" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771317 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2d293b-459f-4f60-9964-c967b95a51e8" containerName="payload-processing" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771357 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2207a59f-da22-43f1-9b51-4d040376d962" containerName="payload-pre-processing" Nov 01 22:06:34.771461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771364 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2207a59f-da22-43f1-9b51-4d040376d962" containerName="payload-pre-processing" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771375 4395 cpu_manager.go:414] "containerMap: removing container" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.771752 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772007 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2207a59f-da22-43f1-9b51-4d040376d962" containerName="payload-pre-processing" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772043 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a39204-5e0e-4a1e-a778-042361d17445" containerName="istio-proxy" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772056 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772080 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e74a22-77dd-4ad1-aca8-e27ec645426e" containerName="istio-proxy" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772089 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9afd5e-0e61-4b1b-9326-d1c86e4a8410" containerName="maas-api" Nov 01 22:06:34.772550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.772098 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2d293b-459f-4f60-9964-c967b95a51e8" containerName="payload-processing" Nov 01 22:06:34.812399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.812368 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:06:34.812565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.812473 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.872743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.872708 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.873051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.872999 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.873217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.873197 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fpf\" (UniqueName: \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.873324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.873310 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.928052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.916570 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d0a5de-serving-cert\"" Nov 01 22:06:34.983053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.979666 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fpf\" (UniqueName: \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.983053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.979772 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.983053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.979822 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:34.997509 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:34.996197 4395 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-serving-cert: secret "maas-api-e2e-inf-a-d0a5de-serving-cert" not found Nov 01 22:06:35.019050 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:34.998123 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls podName:4386dc80-d310-4fcd-8956-5085a17c1f2f nodeName:}" failed. No retries permitted until 2026-11-01 22:06:35.498091795 +0000 UTC m=+1342.471994319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls") pod "maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" (UID: "4386dc80-d310-4fcd-8956-5085a17c1f2f") : secret "maas-api-e2e-inf-a-d0a5de-serving-cert" not found Nov 01 22:06:35.019050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.998184 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.019050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:34.998997 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.019050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:35.017699 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.039529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:35.032787 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fpf\" (UniqueName: \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.514055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:35.509672 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.515122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:35.515092 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:35.732736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:35.732702 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:36.054120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:36.048971 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:06:36.054120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:36.051247 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:06:36.110072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:36.109344 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" event={"ID":"4386dc80-d310-4fcd-8956-5085a17c1f2f","Type":"ContainerStarted","Data":"c543d192e840b39c58131f7e29afd14b512f0f7dc4f4e2ea033bbded995dc08f"} Nov 01 22:06:37.119664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:37.119618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" event={"ID":"4386dc80-d310-4fcd-8956-5085a17c1f2f","Type":"ContainerStarted","Data":"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249"} Nov 01 22:06:37.120135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:37.119848 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:37.144255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:37.144201 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" podStartSLOduration=2.615362851 podStartE2EDuration="3.144183337s" totalImagesPullingTime="528.820486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:37.141869841 +0000 UTC m=+1344.115772596" watchObservedRunningTime="2026-11-01 22:06:37.144183337 +0000 UTC m=+1344.118085875" Nov 01 22:06:39.138920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.138874 4395 generic.go:356] "Generic (PLEG): container finished" podID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerID="b6df81d7b852d637b8f2fc2ebd5439df41fc95cd8beb3b3b30f1f15998741023" exitCode=0 Nov 01 22:06:39.139900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.138997 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" event={"ID":"77599b7b-6134-4adf-8462-bde0e3d20b24","Type":"ContainerDied","Data":"b6df81d7b852d637b8f2fc2ebd5439df41fc95cd8beb3b3b30f1f15998741023"} Nov 01 22:06:39.237737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.237714 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 22:06:39.372816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.372779 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.372834 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373001 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373104 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373204 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373246 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373342 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373420 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373496 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:39.373866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373597 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\") pod \"77599b7b-6134-4adf-8462-bde0e3d20b24\" (UID: \"77599b7b-6134-4adf-8462-bde0e3d20b24\") " Nov 01 22:06:39.373866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.373639 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:39.374151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.374115 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:39.374325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.374306 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.374385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.374330 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.374385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.374345 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77599b7b-6134-4adf-8462-bde0e3d20b24-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.374673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.374650 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:39.375133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.375088 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:39.375647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.375604 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:06:39.376671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.376646 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:06:39.376671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.376656 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:39.377775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.377751 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v" pod "77599b7b-6134-4adf-8462-bde0e3d20b24" (UID: "77599b7b-6134-4adf-8462-bde0e3d20b24"). InnerVolumeSpecName "kube-api-access-p4c6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477076 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477138 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477155 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477182 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477196 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77599b7b-6134-4adf-8462-bde0e3d20b24-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:39.486127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:39.477213 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4c6v\" (UniqueName: \"kubernetes.io/projected/77599b7b-6134-4adf-8462-bde0e3d20b24-kube-api-access-p4c6v\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:40.226101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:40.224955 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" event={"ID":"77599b7b-6134-4adf-8462-bde0e3d20b24","Type":"ContainerDied","Data":"38b90a8ee4ba3efa2bd11b405e3f6dea88b12c83b63943517004eaab8fbbbcd0"} Nov 01 22:06:40.226101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:40.225050 4395 scope.go:122] "RemoveContainer" containerID="b6df81d7b852d637b8f2fc2ebd5439df41fc95cd8beb3b3b30f1f15998741023" Nov 01 22:06:40.226101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:40.225286 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn" Nov 01 22:06:40.318140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:40.318056 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 22:06:40.383620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:40.383587 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bf7xn"] Nov 01 22:06:41.091355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.091319 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:06:41.092451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092424 4395 cpu_manager.go:414] "containerMap: removing container" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" Nov 01 22:06:41.092451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092452 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" Nov 01 22:06:41.092911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092494 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:41.092911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092503 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:41.092911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092719 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea0c9e9-ce17-413a-9efa-feaaf0f57802" containerName="istio-proxy" Nov 01 22:06:41.092911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.092740 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" containerName="istio-proxy" Nov 01 22:06:41.097591 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.097554 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.102665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.102633 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17957f-openshift-default-dockercfg-5jkrs\"" Nov 01 22:06:41.115375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.115352 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:06:41.168810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.168785 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77599b7b-6134-4adf-8462-bde0e3d20b24" path="/var/lib/kubelet/pods/77599b7b-6134-4adf-8462-bde0e3d20b24/volumes" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.316682 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.316808 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.316855 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.316909 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.316982 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.317050 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndr8\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.317101 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.317135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.327066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.317343 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.333706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.333673 4395 generic.go:356] "Generic (PLEG): container finished" podID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerID="10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc" exitCode=0 Nov 01 22:06:41.334207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.334177 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerDied","Data":"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc"} Nov 01 22:06:41.334936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.334914 4395 scope.go:122] "RemoveContainer" containerID="10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc" Nov 01 22:06:41.361067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.346824 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:41.361067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.346928 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:41.361067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.346973 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440471 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440589 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440656 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440715 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndr8\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440761 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440792 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.440950 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.441105 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.441633 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.450425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.441881 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.490196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.482565 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.490196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.488783 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.490196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.489605 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.503168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.503110 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndr8\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.509537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.509434 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.509537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.509470 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.525814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.525008 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:41.718341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:41.718285 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:42.114049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.111287 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:42.114049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.111384 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:42.114049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.111444 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:42.114049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.112093 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:06:42.145545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.145514 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:06:42.151514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.151495 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.164328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.164303 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:06:42.221374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.221349 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3577a0-openshift-default-dockercfg-nmmqq\"" Nov 01 22:06:42.291203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291165 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9lw\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291335 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291453 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291489 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291553 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291680 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.291812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.291798 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.373264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.373181 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerStarted","Data":"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262"} Nov 01 22:06:42.373264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.373241 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerStarted","Data":"92f38979cbaca9f9627dd5bea6fbefdc996acb00baabefac2e312ebc07e907d9"} Nov 01 22:06:42.375420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.375395 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerStarted","Data":"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569"} Nov 01 22:06:42.393368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393340 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393395 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393448 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393621 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9lw\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393666 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393697 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393744 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393768 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393751 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393838 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393897 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.393987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.393937 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.394928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.394901 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.395123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.395102 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.397002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.396982 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.397250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.397232 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.397776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.397741 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podStartSLOduration=1.397730271 podStartE2EDuration="1.397730271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:42.396099546 +0000 UTC m=+1349.370002084" watchObservedRunningTime="2026-11-01 22:06:42.397730271 +0000 UTC m=+1349.371632810" Nov 01 22:06:42.408007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.407981 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.408179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.408157 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9lw\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.505050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.504308 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:42.547479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.547449 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:06:42.547826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.547736 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:42.548078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.548053 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:42.721149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:42.719276 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:43.056080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.053809 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:06:43.059242 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:06:43.059211 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb96a1f_ea6e_446a_b60a_cccccc414a3b.slice/crio-982e9fee8cceac454c21855baa6c2f16d9ccc9e9c962950798dcdf14df56a0de WatchSource:0}: Error finding container 982e9fee8cceac454c21855baa6c2f16d9ccc9e9c962950798dcdf14df56a0de: Status 404 returned error can't find the container with id 982e9fee8cceac454c21855baa6c2f16d9ccc9e9c962950798dcdf14df56a0de Nov 01 22:06:43.066236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.064394 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:43.066236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.064544 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:43.066236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.064612 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:43.135378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.135353 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:06:43.416315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.416225 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" event={"ID":"cfb96a1f-ea6e-446a-b60a-cccccc414a3b","Type":"ContainerStarted","Data":"03b2ebfdc40ff5e5d22ab94e2e9e8f88c11390db2ee88ec008ca7cd2ff7835a0"} Nov 01 22:06:43.416776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.416309 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" event={"ID":"cfb96a1f-ea6e-446a-b60a-cccccc414a3b","Type":"ContainerStarted","Data":"982e9fee8cceac454c21855baa6c2f16d9ccc9e9c962950798dcdf14df56a0de"} Nov 01 22:06:43.444823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.444771 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podStartSLOduration=1.4447578669999999 podStartE2EDuration="1.444757867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:43.440981737 +0000 UTC m=+1350.414884278" watchObservedRunningTime="2026-11-01 22:06:43.444757867 +0000 UTC m=+1350.418660406" Nov 01 22:06:43.505679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.505644 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:43.532088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.532058 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:43.532277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.532113 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:43.725207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:43.725123 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:44.336688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.335938 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:06:44.463906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.463874 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:44.466002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.465692 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:06:44.552401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.551227 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:44.552401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.551284 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:44.552401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.551745 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:44.552401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:44.551776 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:45.510059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:45.509528 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:45.510059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:45.509586 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:45.532185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:45.532153 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:45.532347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:45.532222 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:46.425056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.424083 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:06:46.436557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.433341 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.491053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.461724 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9nb\" (UniqueName: \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.491053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.461782 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.527155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.509639 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-17957f-dockercfg-l6gx2\"" Nov 01 22:06:46.527155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.509996 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:46.527155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.510071 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:46.557054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.554186 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:46.557054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.554255 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:46.622278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.585186 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9nb\" (UniqueName: \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.622278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.585260 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.657078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.654729 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17957f\"" Nov 01 22:06:46.700146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.692265 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.861052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.846885 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9nb\" (UniqueName: \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\") pod \"payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:46.901534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.892955 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:06:46.946342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.946104 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:06:46.985653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.985566 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.011055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.987216 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:06:47.011055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:46.994346 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:47.027834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.027807 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edd32a-openshift-default-dockercfg-lkssj\"" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.030521 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041345 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041414 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwfn\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041446 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041481 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041529 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041679 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041820 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.042290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.041856 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143742 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143813 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143869 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwfn\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143904 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143935 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.143980 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.144148 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.144283 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.144764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.144320 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.145531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.144952 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.145531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.145446 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.146388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.145801 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.146388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.146006 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.146753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.146728 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.148170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.148149 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.148326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.148298 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.160663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.160632 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.160769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.160743 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwfn\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.509668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.508442 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:47.534054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.529526 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:47.534054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.529584 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:47.534772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.534438 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:47.534772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.534486 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:47.541067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.541008 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:06:47.861944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:47.861804 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" event={"ID":"6d051c77-396c-435b-bdac-f5033e0dde02","Type":"ContainerStarted","Data":"0291744e0cbe1dfa026171cbadffb43cccbc78cf9e0666fd7c767f58b8adb134"} Nov 01 22:06:48.342207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.342176 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:06:48.343341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.343308 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:06:48.345991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.345957 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:48.346112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.346049 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:48.346112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.346095 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:06:48.511052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.505422 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:48.511052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.505482 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:48.532936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.532907 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:06:48.533117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.532976 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 01 22:06:48.872532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.872433 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" event={"ID":"6d051c77-396c-435b-bdac-f5033e0dde02","Type":"ContainerStarted","Data":"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9"} Nov 01 22:06:48.873195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.873164 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:06:48.882061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.881724 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" event={"ID":"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45","Type":"ContainerStarted","Data":"df143d6d6101e44c98f46c8e54f31fe502de1ed66ecbf2c623a8521f3b87f765"} Nov 01 22:06:48.882061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.881785 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" event={"ID":"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45","Type":"ContainerStarted","Data":"28eb0eae96fc8d2a390138a713b4018659b3fc4e40327244e068f136f072c775"} Nov 01 22:06:48.920871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:48.919996 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" podStartSLOduration=2.9199779489999997 podStartE2EDuration="2.919977949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:48.91760155 +0000 UTC m=+1355.891504093" watchObservedRunningTime="2026-11-01 22:06:48.919977949 +0000 UTC m=+1355.893880490" Nov 01 22:06:49.107962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.107550 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podStartSLOduration=3.107528161 podStartE2EDuration="3.107528161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:48.960482609 +0000 UTC m=+1355.934385174" watchObservedRunningTime="2026-11-01 22:06:49.107528161 +0000 UTC m=+1356.081430701" Nov 01 22:06:49.337274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.337208 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" podUID="5014d6cb-8d99-4691-932f-f8c0ca090cea" containerName="payload-pre-processing" containerID="cri-o://2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838" gracePeriod=25 Nov 01 22:06:49.505180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.505146 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.505359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.505201 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:49.517425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.517355 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:49.519890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.519758 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:49.519890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.519822 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:49.538677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.538645 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:06:49.540061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.540002 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:06:49.847963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.847668 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:06:49.859664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.859635 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\") pod \"5014d6cb-8d99-4691-932f-f8c0ca090cea\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " Nov 01 22:06:49.859910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.859836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\" (UniqueName: \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\") pod \"5014d6cb-8d99-4691-932f-f8c0ca090cea\" (UID: \"5014d6cb-8d99-4691-932f-f8c0ca090cea\") " Nov 01 22:06:49.860601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.860550 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume" pod "5014d6cb-8d99-4691-932f-f8c0ca090cea" (UID: "5014d6cb-8d99-4691-932f-f8c0ca090cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:06:49.861021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.861000 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5014d6cb-8d99-4691-932f-f8c0ca090cea-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:49.864823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.864799 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7" pod "5014d6cb-8d99-4691-932f-f8c0ca090cea" (UID: "5014d6cb-8d99-4691-932f-f8c0ca090cea"). InnerVolumeSpecName "kube-api-access-4s2z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:06:49.934998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.934962 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" Nov 01 22:06:49.936055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.936002 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" event={"ID":"5014d6cb-8d99-4691-932f-f8c0ca090cea","Type":"ContainerDied","Data":"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838"} Nov 01 22:06:49.936256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.936237 4395 scope.go:122] "RemoveContainer" containerID="2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838" Nov 01 22:06:49.936469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.936168 4395 generic.go:356] "Generic (PLEG): container finished" podID="5014d6cb-8d99-4691-932f-f8c0ca090cea" containerID="2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838" exitCode=0 Nov 01 22:06:49.937115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.937067 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g" event={"ID":"5014d6cb-8d99-4691-932f-f8c0ca090cea","Type":"ContainerDied","Data":"9337ce6401367a5a59e1e0489ee2924419a89f8c1e8dbfb11a40c4acec2ab399"} Nov 01 22:06:49.953357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.953335 4395 scope.go:122] "RemoveContainer" containerID="2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838" Nov 01 22:06:49.953770 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:49.953718 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838\": container with ID starting with 2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838 not found: ID does not exist" containerID="2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838" Nov 01 22:06:49.953910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.953889 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838"} err="failed to get container status \"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838\": rpc error: code = NotFound desc = could not find container \"2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838\": container with ID starting with 2d223aed34614f158594093ddf1aea843e8a4e9220613cb38f018653484bf838 not found: ID does not exist" Nov 01 22:06:49.988108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:49.968827 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2z7\" (UniqueName: \"kubernetes.io/projected/5014d6cb-8d99-4691-932f-f8c0ca090cea-kube-api-access-4s2z7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:06:50.064345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.064312 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:06:50.184072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.177163 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bb882c-c9cf8d987-ljp4g"] Nov 01 22:06:50.505011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.504972 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.505252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.505055 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 01 22:06:50.517149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.517123 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:50.517320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:50.517169 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:51.167874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.167841 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5014d6cb-8d99-4691-932f-f8c0ca090cea" path="/var/lib/kubelet/pods/5014d6cb-8d99-4691-932f-f8c0ca090cea/volumes" Nov 01 22:06:51.510790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.510763 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:51.511077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.511059 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:51.511885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.511868 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:06:51.516830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.516801 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:51.516929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:51.516839 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:52.347212 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:06:52.347171 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:06:52.517081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:52.517046 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:52.517266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:52.517117 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:53.520616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:53.520579 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:53.521240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:53.521209 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:54.516920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:54.516883 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:54.517108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:54.516957 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:55.516383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:55.516332 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:06:55.516897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:55.516407 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:06:56.521933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:56.521898 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:56.522450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:56.522290 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:06:56.523130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:06:56.523110 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:07:00.973370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:00.973344 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:07:02.035632 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.035596 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:07:02.036211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.035838 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" podUID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" containerName="maas-api" containerID="cri-o://0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d" gracePeriod=30 Nov 01 22:07:02.746876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.746853 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:07:02.863225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.863190 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\") pod \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " Nov 01 22:07:02.863425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.863278 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\") pod \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " Nov 01 22:07:02.863425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.863389 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\" (UniqueName: \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\") pod \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " Nov 01 22:07:02.863562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.863545 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\") pod \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\" (UID: \"8a8c9693-c252-413c-a0e9-ba452ef8a42a\") " Nov 01 22:07:02.863714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.863683 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca" pod "8a8c9693-c252-413c-a0e9-ba452ef8a42a" (UID: "8a8c9693-c252-413c-a0e9-ba452ef8a42a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:07:02.864475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.864449 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8c9693-c252-413c-a0e9-ba452ef8a42a-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:02.865935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.865903 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs" pod "8a8c9693-c252-413c-a0e9-ba452ef8a42a" (UID: "8a8c9693-c252-413c-a0e9-ba452ef8a42a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:07:02.867772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.867744 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls" pod "8a8c9693-c252-413c-a0e9-ba452ef8a42a" (UID: "8a8c9693-c252-413c-a0e9-ba452ef8a42a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:07:02.868275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.868248 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk" pod "8a8c9693-c252-413c-a0e9-ba452ef8a42a" (UID: "8a8c9693-c252-413c-a0e9-ba452ef8a42a"). InnerVolumeSpecName "kube-api-access-mc8mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:02.965865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.965835 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8mk\" (UniqueName: \"kubernetes.io/projected/8a8c9693-c252-413c-a0e9-ba452ef8a42a-kube-api-access-mc8mk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:02.965865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.965862 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:02.965865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:02.965871 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a8c9693-c252-413c-a0e9-ba452ef8a42a-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:03.157875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.157840 4395 generic.go:356] "Generic (PLEG): container finished" podID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" containerID="0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d" exitCode=0 Nov 01 22:07:03.162161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.162128 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" Nov 01 22:07:03.168559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.168533 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" event={"ID":"8a8c9693-c252-413c-a0e9-ba452ef8a42a","Type":"ContainerDied","Data":"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d"} Nov 01 22:07:03.168674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.168581 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d" event={"ID":"8a8c9693-c252-413c-a0e9-ba452ef8a42a","Type":"ContainerDied","Data":"27dfe02e55fad896ca653bdbf7d969fab21f519d1c6f38e631d3573562ae196d"} Nov 01 22:07:03.168674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.168603 4395 scope.go:122] "RemoveContainer" containerID="0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d" Nov 01 22:07:03.181135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.181111 4395 scope.go:122] "RemoveContainer" containerID="0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d" Nov 01 22:07:03.181470 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:07:03.181437 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d\": container with ID starting with 0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d not found: ID does not exist" containerID="0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d" Nov 01 22:07:03.181542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.181473 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d"} err="failed to get container status \"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d\": rpc error: code = NotFound desc = could not find container \"0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d\": container with ID starting with 0edb1470ca29972c3b87b236be046f067e42b1479bb73269a846b5cd26ec2a8d not found: ID does not exist" Nov 01 22:07:03.196885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.196854 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:07:03.200801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:03.200782 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0390e1-564fbc7f96-52l5d"] Nov 01 22:07:05.257084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.256684 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" path="/var/lib/kubelet/pods/8a8c9693-c252-413c-a0e9-ba452ef8a42a/volumes" Nov 01 22:07:05.495331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.495294 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:07:05.497323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497294 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" containerName="maas-api" Nov 01 22:07:05.497485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497332 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" containerName="maas-api" Nov 01 22:07:05.497540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497486 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5014d6cb-8d99-4691-932f-f8c0ca090cea" containerName="payload-pre-processing" Nov 01 22:07:05.497540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497495 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5014d6cb-8d99-4691-932f-f8c0ca090cea" containerName="payload-pre-processing" Nov 01 22:07:05.497900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497879 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5014d6cb-8d99-4691-932f-f8c0ca090cea" containerName="payload-pre-processing" Nov 01 22:07:05.497999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.497913 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8c9693-c252-413c-a0e9-ba452ef8a42a" containerName="maas-api" Nov 01 22:07:05.508087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.507984 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.511992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.511959 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:07:05.514692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.514670 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9d8b5d-dockercfg-vz66r\"" Nov 01 22:07:05.516088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.516021 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9d8b5d\"" Nov 01 22:07:05.539769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.539738 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.539905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.539849 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tk9m\" (UniqueName: \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.649441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.648156 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.649441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.648225 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tk9m\" (UniqueName: \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.672060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.669527 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.687053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.678325 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tk9m\" (UniqueName: \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\") pod \"payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:05.840424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:05.840313 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:06.267917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:06.267886 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:07:06.269011 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:07:06.268986 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd516c35b_22c9_44c9_b704_f79b10ff516d.slice/crio-cf2cc552e7b89b3f829755d4a10e20726f77740e5e7bddc16311a839d071a7bf WatchSource:0}: Error finding container cf2cc552e7b89b3f829755d4a10e20726f77740e5e7bddc16311a839d071a7bf: Status 404 returned error can't find the container with id cf2cc552e7b89b3f829755d4a10e20726f77740e5e7bddc16311a839d071a7bf Nov 01 22:07:07.304054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:07.292805 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" event={"ID":"d516c35b-22c9-44c9-b704-f79b10ff516d","Type":"ContainerStarted","Data":"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276"} Nov 01 22:07:07.304054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:07.292900 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" event={"ID":"d516c35b-22c9-44c9-b704-f79b10ff516d","Type":"ContainerStarted","Data":"cf2cc552e7b89b3f829755d4a10e20726f77740e5e7bddc16311a839d071a7bf"} Nov 01 22:07:07.304054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:07.294208 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:14.204947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.204880 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" podStartSLOduration=9.204860034 podStartE2EDuration="9.204860034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:07.344482689 +0000 UTC m=+1374.318385231" watchObservedRunningTime="2026-11-01 22:07:14.204860034 +0000 UTC m=+1381.178762574" Nov 01 22:07:14.207565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.207535 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:07:14.213080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.213011 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.220593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.220548 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0f9b7e\"" Nov 01 22:07:14.223572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.223538 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:07:14.246886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.246849 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0f9b7e-dockercfg-9mt99\"" Nov 01 22:07:14.320328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.320282 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.331057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.320716 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.452054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.445138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.452054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.445263 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.452054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.446469 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.569879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.569845 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\") pod \"payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:14.842529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:14.842444 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:15.170842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:15.170819 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:07:15.641020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:15.640974 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" event={"ID":"af57c9eb-c623-4c3b-9ac9-73b08c696ddc","Type":"ContainerStarted","Data":"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39"} Nov 01 22:07:15.641520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:15.641066 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" event={"ID":"af57c9eb-c623-4c3b-9ac9-73b08c696ddc","Type":"ContainerStarted","Data":"35edbf19e2ab4e7848cd1a77b0bfe216a007ba3a94784d5c9eab1c96d4496726"} Nov 01 22:07:15.641520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:15.641101 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:15.662671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:15.662585 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" podStartSLOduration=1.66257154 podStartE2EDuration="1.66257154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:15.661135351 +0000 UTC m=+1382.635037887" watchObservedRunningTime="2026-11-01 22:07:15.66257154 +0000 UTC m=+1382.636474079" Nov 01 22:07:16.663138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.663102 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:07:16.668568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.668545 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:16.685543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.680107 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:07:16.774970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.769465 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f2760d-dockercfg-t8q6k\"" Nov 01 22:07:16.774970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.774670 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f2760d\"" Nov 01 22:07:16.938115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.934161 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:16.938115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:16.934391 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlxd\" (UniqueName: \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.037813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.037768 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.038007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.037973 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlxd\" (UniqueName: \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.049280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.039629 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.056364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.056332 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlxd\" (UniqueName: \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\") pod \"payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.291578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.291538 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:17.580020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.579990 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:07:17.580171 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:07:17.580102 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c54621_69e3_4946_9ef9_ccf05fff0199.slice/crio-e8484751bbafade7e252e46899977352411ffe7500d55ddaf3790a12b513ebca WatchSource:0}: Error finding container e8484751bbafade7e252e46899977352411ffe7500d55ddaf3790a12b513ebca: Status 404 returned error can't find the container with id e8484751bbafade7e252e46899977352411ffe7500d55ddaf3790a12b513ebca Nov 01 22:07:17.663328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:17.663289 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" event={"ID":"a2c54621-69e3-4946-9ef9-ccf05fff0199","Type":"ContainerStarted","Data":"e8484751bbafade7e252e46899977352411ffe7500d55ddaf3790a12b513ebca"} Nov 01 22:07:18.757442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:18.757400 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" event={"ID":"a2c54621-69e3-4946-9ef9-ccf05fff0199","Type":"ContainerStarted","Data":"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43"} Nov 01 22:07:18.758961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:18.758930 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:19.392225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:19.392189 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:07:19.413101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:19.413049 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" podStartSLOduration=3.413015286 podStartE2EDuration="3.413015286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:18.779927705 +0000 UTC m=+1385.753830245" watchObservedRunningTime="2026-11-01 22:07:19.413015286 +0000 UTC m=+1386.386917881" Nov 01 22:07:20.622690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:20.622640 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:07:20.623475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:20.622882 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerName="maas-api" containerID="cri-o://1ab433615970f440e64cdb3d30ce125e6eff56806a46d75bfc2cb4685715c02f" gracePeriod=30 Nov 01 22:07:20.774048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:20.773992 4395 generic.go:356] "Generic (PLEG): container finished" podID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerID="1ab433615970f440e64cdb3d30ce125e6eff56806a46d75bfc2cb4685715c02f" exitCode=0 Nov 01 22:07:20.774218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:20.774077 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" event={"ID":"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4","Type":"ContainerDied","Data":"1ab433615970f440e64cdb3d30ce125e6eff56806a46d75bfc2cb4685715c02f"} Nov 01 22:07:21.349065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.348479 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:07:21.449337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.449253 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\" (UniqueName: \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\") pod \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " Nov 01 22:07:21.449337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.449295 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\") pod \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " Nov 01 22:07:21.449550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.449446 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\") pod \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " Nov 01 22:07:21.449624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.449601 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\") pod \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\" (UID: \"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4\") " Nov 01 22:07:21.449691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.449625 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca" pod "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" (UID: "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:07:21.450047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.450014 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:21.451680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.451655 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls" pod "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" (UID: "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:07:21.451793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.451709 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x" pod "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" (UID: "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4"). InnerVolumeSpecName "kube-api-access-xzp6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:21.451856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.451837 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs" pod "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" (UID: "dfc7ea1f-a814-4a35-b5a3-2b7df00983c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:07:21.551343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.551307 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:21.551343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.551335 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzp6x\" (UniqueName: \"kubernetes.io/projected/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-kube-api-access-xzp6x\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:21.551343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.551346 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:21.783749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.783721 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" Nov 01 22:07:21.784226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.783720 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" event={"ID":"dfc7ea1f-a814-4a35-b5a3-2b7df00983c4","Type":"ContainerDied","Data":"819f85f7747b04cdc188fde84d04a3e29becfac3d731c9f888a7deadc5ae2a0f"} Nov 01 22:07:21.784226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.783880 4395 scope.go:122] "RemoveContainer" containerID="1ab433615970f440e64cdb3d30ce125e6eff56806a46d75bfc2cb4685715c02f" Nov 01 22:07:21.813415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.813383 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:07:21.817232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:21.817212 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx"] Nov 01 22:07:22.200714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.200627 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:07:22.201773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.201749 4395 cpu_manager.go:414] "containerMap: removing container" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerName="maas-api" Nov 01 22:07:22.201773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.201773 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerName="maas-api" Nov 01 22:07:22.201983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.201947 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerName="maas-api" Nov 01 22:07:22.212168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.209782 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.213456 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.213428 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:07:22.221161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.221139 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9d8b5d-serving-cert\"" Nov 01 22:07:22.260838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.260803 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.261062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.261042 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrhf\" (UniqueName: \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.261293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.261270 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.261414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.261398 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.304535 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:07:22.304494 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:07:22.373052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.367624 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrhf\" (UniqueName: \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.373052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.368367 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.373052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.369061 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.373052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.369626 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.373052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.370339 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.409984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.401618 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.409984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.403561 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.432904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.430856 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrhf\" (UniqueName: \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\") pod \"maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.530707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.530679 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:22.934096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:22.925251 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:07:23.167539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:23.167469 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" path="/var/lib/kubelet/pods/dfc7ea1f-a814-4a35-b5a3-2b7df00983c4/volumes" Nov 01 22:07:23.827655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:23.827621 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" event={"ID":"2db0c166-47aa-4aa8-be5d-f949cbddfbd7","Type":"ContainerStarted","Data":"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304"} Nov 01 22:07:23.827826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:23.827674 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" event={"ID":"2db0c166-47aa-4aa8-be5d-f949cbddfbd7","Type":"ContainerStarted","Data":"f9c69636a4531e0c949a6cfd3bc8d679c9b5fef921c86350f99061adbf7c55a0"} Nov 01 22:07:23.827826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:23.827695 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:23.852432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:23.852376 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" podStartSLOduration=1.300204176 podStartE2EDuration="1.852360535s" totalImagesPullingTime="552.156359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:23.847578855 +0000 UTC m=+1390.821481395" watchObservedRunningTime="2026-11-01 22:07:23.852360535 +0000 UTC m=+1390.826263091" Nov 01 22:07:24.256106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:24.246794 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb882c-8f6cd6d4b-rv7dx" podUID="dfc7ea1f-a814-4a35-b5a3-2b7df00983c4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.121:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:26.341751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.341703 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:07:26.379055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.371086 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.379055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.376506 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508073 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508164 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508229 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508362 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508388 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508462 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508513 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcmc\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508569 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508601 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.524068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.508870 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-830149-openshift-default-dockercfg-kkt65\"" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622446 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622607 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622690 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622763 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.622952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.623061 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.623118 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcmc\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.745994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.721730 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.762439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.749339 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:07:26.818350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.818315 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcmc\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.847056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.837351 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:07:26.847056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.846183 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.852411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.852377 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.853837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.852908 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.854676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.854656 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.879057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.860529 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.879057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.861924 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.947592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.946672 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:26.978402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:26.976975 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:27.003125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.001891 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:27.033064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.030881 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:27.075064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.060598 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:27.075064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.070166 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:27.656347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.654929 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:07:27.660413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.658078 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:27.660413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.658157 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:27.660413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:27.658198 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:28.389055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:28.383407 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerStarted","Data":"f98bde87b5f0bf246ce431c7235a1fe5003a65cb1b215cf2346dc80abf9681aa"} Nov 01 22:07:28.389055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:28.383480 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerStarted","Data":"d0d710494ab0c71239f2de2934740ac36bad183e7ff3355ae9858f38702d63ec"} Nov 01 22:07:28.483053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:28.465574 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podStartSLOduration=2.465552595 podStartE2EDuration="2.465552595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:28.455779545 +0000 UTC m=+1395.429682086" watchObservedRunningTime="2026-11-01 22:07:28.465552595 +0000 UTC m=+1395.439455136" Nov 01 22:07:28.989055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:28.971133 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.038172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.038120 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.084055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.081200 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.084055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.082758 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:29.084055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.083823 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:07:29.084055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.083885 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:07:29.363172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.362143 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.414312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.414224 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.533889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.533837 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.582207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.582163 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.988233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.988188 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:29.995468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:29.988839 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.056864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.056824 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.068363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.057530 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:07:30.172569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.101782 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.172569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.101857 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:07:30.172569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.164188 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:07:30.285063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.283251 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:07:30.305434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.305402 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:30.429052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.403002 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:07:30.471908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.461466 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:07:30.471908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.462961 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:07:30.609202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.584671 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.609202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.585254 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:07:30.819282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.819247 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:07:30.890518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.888406 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:30.916868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.916841 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:07:30.985086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.983099 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:30.998322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:30.998287 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:07:31.000160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.000134 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:07:31.050721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.048794 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:07:31.050721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.049262 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:07:31.094452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.094412 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 01 22:07:31.096453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.096418 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:07:31.433157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.433114 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:31.470255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.470215 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:07:31.532168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.532121 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:31.597439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.597296 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:31.721093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.704912 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:07:31.733294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.732134 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 22:07:31.835847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.835812 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:07:31.862708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.857161 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997441 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997519 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997608 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997731 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997775 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997840 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997887 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.011117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:31.997969 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvmf\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.016052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.012131 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:32.049119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.040166 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:07:32.050709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.050680 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:32.055999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.055970 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:07:32.070249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.070201 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:07:32.070811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.070778 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:07:32.157192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157147 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157247 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157341 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157462 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157556 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157592 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157641 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.157774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.157725 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvmf\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.158812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.158772 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.158952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.158823 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:07:32.159140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.159118 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.172322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.171769 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.172322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.172087 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.173461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.172725 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:07:32.180085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.174253 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.246467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.246436 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:07:32.254191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.249825 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.254191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.251043 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvmf\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.254191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.252485 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.258724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.258606 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.327104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.326976 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:07:32.350750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.350718 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:32.755718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.755689 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:07:32.763389 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:07:32.763356 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a30721_9e35_46c5_88c2_47b275f00a7a.slice/crio-a085653746b8b390db92cea6e885a8bc6a76446767a71d0f132eac4956523eb2 WatchSource:0}: Error finding container a085653746b8b390db92cea6e885a8bc6a76446767a71d0f132eac4956523eb2: Status 404 returned error can't find the container with id a085653746b8b390db92cea6e885a8bc6a76446767a71d0f132eac4956523eb2 Nov 01 22:07:32.766520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.766490 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:32.766601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.766558 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:32.766601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:32.766586 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:33.389056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:33.380498 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" event={"ID":"51a30721-9e35-46c5-88c2-47b275f00a7a","Type":"ContainerStarted","Data":"81a0ca7b2b7806acb95f5b36029f5995dc65eead6784739e64a911180ea27934"} Nov 01 22:07:33.389056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:33.380554 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" event={"ID":"51a30721-9e35-46c5-88c2-47b275f00a7a","Type":"ContainerStarted","Data":"a085653746b8b390db92cea6e885a8bc6a76446767a71d0f132eac4956523eb2"} Nov 01 22:07:33.494368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:33.494277 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podStartSLOduration=2.494255806 podStartE2EDuration="2.494255806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:33.451472064 +0000 UTC m=+1400.425374604" watchObservedRunningTime="2026-11-01 22:07:33.494255806 +0000 UTC m=+1400.468158347" Nov 01 22:07:34.351391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:34.351359 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:34.353207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:34.353183 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:34.353316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:34.353232 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:35.350968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:35.350934 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:35.351383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:35.350991 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:36.351059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:36.350999 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:36.351530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:36.351111 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:37.351377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:37.351342 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:37.351858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:37.351415 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:38.351169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:38.351137 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:38.351380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:38.351196 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:39.352967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:39.352274 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:39.352967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:39.352351 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:40.051693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.051647 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:07:40.059247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.059207 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.066113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.066085 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f2760d-serving-cert\"" Nov 01 22:07:40.070858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.070779 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:07:40.142984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.142945 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfz4j\" (UniqueName: \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.143234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.142999 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.143234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.143142 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.143366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.143292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.250320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.249987 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfz4j\" (UniqueName: \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.250320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.250144 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.252542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.252512 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.277747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.271240 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.277747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.271619 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.372733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.369597 4395 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Nov 01 22:07:40.372733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.369677 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Nov 01 22:07:40.372733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.371203 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.425331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.402767 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfz4j\" (UniqueName: \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.425331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.419887 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\") pod \"maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.723452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.717310 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:40.828803 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:40.828768 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:07:41.392152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.392118 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:07:41.462733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.461483 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:41.491344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.491304 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:41.491520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.491428 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:07:41.555200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.555079 4395 generic.go:356] "Generic (PLEG): container finished" podID="7642982a-fd39-4afc-b878-798249a7a0cd" containerID="10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262" exitCode=0 Nov 01 22:07:41.555820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.555590 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerDied","Data":"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262"} Nov 01 22:07:41.558608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.558588 4395 scope.go:122] "RemoveContainer" containerID="10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262" Nov 01 22:07:41.580110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.580008 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:41.599794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.599430 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:41.599794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.599502 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:07:41.607645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:41.607622 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" event={"ID":"44424331-241a-4818-aecd-b429fec79810","Type":"ContainerStarted","Data":"0be004cff457b498cb706357099c19108229af9c2da6029715d7c26eeab01d05"} Nov 01 22:07:42.624020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.623981 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" event={"ID":"44424331-241a-4818-aecd-b429fec79810","Type":"ContainerStarted","Data":"86a157db20ceed175a7bc1ed10a2efd2aacc83036b79fc384532ca6c07cca3ba"} Nov 01 22:07:42.624490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.624139 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:42.626666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.626638 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerStarted","Data":"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3"} Nov 01 22:07:42.644991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.644931 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" podStartSLOduration=2.161876173 podStartE2EDuration="2.64491157s" totalImagesPullingTime="483.035397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:42.643410499 +0000 UTC m=+1409.617313061" watchObservedRunningTime="2026-11-01 22:07:42.64491157 +0000 UTC m=+1409.618814138" Nov 01 22:07:42.718718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.718682 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:07:42.719634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.719582 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:42.719766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:42.719642 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:43.718898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:43.718866 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:43.719321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:43.718927 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:44.463314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:44.463277 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:07:44.719477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:44.719388 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:44.719477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:44.719440 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:45.719349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:45.719316 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:45.719529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:45.719373 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:45.837945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:45.836175 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" podUID="7330bb1f-371f-478e-af31-3e83f7ec5704" containerName="payload-processing" containerID="cri-o://caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611" gracePeriod=25 Nov 01 22:07:46.233462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.233400 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:07:46.346957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.346925 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\" (UniqueName: \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\") pod \"7330bb1f-371f-478e-af31-3e83f7ec5704\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " Nov 01 22:07:46.347192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.347060 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\") pod \"7330bb1f-371f-478e-af31-3e83f7ec5704\" (UID: \"7330bb1f-371f-478e-af31-3e83f7ec5704\") " Nov 01 22:07:46.347764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.347734 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume" pod "7330bb1f-371f-478e-af31-3e83f7ec5704" (UID: "7330bb1f-371f-478e-af31-3e83f7ec5704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:07:46.348158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.348067 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7330bb1f-371f-478e-af31-3e83f7ec5704-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:46.350228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.350131 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n" pod "7330bb1f-371f-478e-af31-3e83f7ec5704" (UID: "7330bb1f-371f-478e-af31-3e83f7ec5704"). InnerVolumeSpecName "kube-api-access-dzk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:07:46.450297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.450259 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzk4n\" (UniqueName: \"kubernetes.io/projected/7330bb1f-371f-478e-af31-3e83f7ec5704-kube-api-access-dzk4n\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:07:46.733824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.733705 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:46.734455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.734410 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:46.736957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.736931 4395 generic.go:356] "Generic (PLEG): container finished" podID="7330bb1f-371f-478e-af31-3e83f7ec5704" containerID="caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611" exitCode=0 Nov 01 22:07:46.737506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.737464 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" event={"ID":"7330bb1f-371f-478e-af31-3e83f7ec5704","Type":"ContainerDied","Data":"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611"} Nov 01 22:07:46.737723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.737706 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" event={"ID":"7330bb1f-371f-478e-af31-3e83f7ec5704","Type":"ContainerDied","Data":"e08fb80aa35c9477db38e13ffd616c0cb11c9c01ee592e920b2ac2ed296e7011"} Nov 01 22:07:46.737869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.737857 4395 scope.go:122] "RemoveContainer" containerID="caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611" Nov 01 22:07:46.742652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.739927 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6" Nov 01 22:07:46.788602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.788575 4395 scope.go:122] "RemoveContainer" containerID="caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611" Nov 01 22:07:46.792247 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:07:46.792206 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611\": container with ID starting with caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611 not found: ID does not exist" containerID="caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611" Nov 01 22:07:46.796171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.796104 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611"} err="failed to get container status \"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611\": rpc error: code = NotFound desc = could not find container \"caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611\": container with ID starting with caba524e5628223d0000d9956bac7e0b81d1ff56b6c671d981aa2bba022e8611 not found: ID does not exist" Nov 01 22:07:46.837383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.832104 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:07:46.837882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:46.837856 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18657f-865b44ff74-n8hw6"] Nov 01 22:07:47.206378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:47.206343 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7330bb1f-371f-478e-af31-3e83f7ec5704" path="/var/lib/kubelet/pods/7330bb1f-371f-478e-af31-3e83f7ec5704/volumes" Nov 01 22:07:47.721106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:47.719419 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 01 22:07:47.721106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:47.719489 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 01 22:07:48.724107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:48.724075 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:07:48.725342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:48.725320 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:07:49.698672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:49.697415 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:07:52.303682 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:07:52.303642 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:07:52.557719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.557635 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:07:52.562075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.559153 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7330bb1f-371f-478e-af31-3e83f7ec5704" containerName="payload-processing" Nov 01 22:07:52.562075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.559177 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7330bb1f-371f-478e-af31-3e83f7ec5704" containerName="payload-processing" Nov 01 22:07:52.562075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.559394 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7330bb1f-371f-478e-af31-3e83f7ec5704" containerName="payload-processing" Nov 01 22:07:52.565282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.565257 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.579539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.579508 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:07:52.602921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.602899 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-830149-serving-cert\"" Nov 01 22:07:52.674335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.674284 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.674697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.674665 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.675178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.675147 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjvg\" (UniqueName: \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.675381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.675353 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.777387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.777346 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjvg\" (UniqueName: \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.778080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.778053 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.778353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.778330 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.779902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.779824 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.780087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.778896 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.797648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.797099 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjvg\" (UniqueName: \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.797648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.797345 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.806661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.806628 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\") pod \"maas-api-e2e-inf-b-830149-65bcbf6998-x28xv\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:52.890516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:52.890451 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:53.178873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:53.178848 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:07:53.178997 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:07:53.178912 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b69845_1b1d_442a_b32c_3071809ba49f.slice/crio-5b3d0cdd17dac6471e637fa19c295d5daac21eb8e3958c58a17490a5fe9f14a8 WatchSource:0}: Error finding container 5b3d0cdd17dac6471e637fa19c295d5daac21eb8e3958c58a17490a5fe9f14a8: Status 404 returned error can't find the container with id 5b3d0cdd17dac6471e637fa19c295d5daac21eb8e3958c58a17490a5fe9f14a8 Nov 01 22:07:53.926448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:53.926344 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" event={"ID":"a4b69845-1b1d-442a-b32c-3071809ba49f","Type":"ContainerStarted","Data":"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe"} Nov 01 22:07:53.926448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:53.926413 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" event={"ID":"a4b69845-1b1d-442a-b32c-3071809ba49f","Type":"ContainerStarted","Data":"5b3d0cdd17dac6471e637fa19c295d5daac21eb8e3958c58a17490a5fe9f14a8"} Nov 01 22:07:53.926448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:53.926447 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:07:53.949324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:53.949273 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" podStartSLOduration=1.522052535 podStartE2EDuration="1.949258731s" totalImagesPullingTime="427.206196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:53.946314021 +0000 UTC m=+1420.920216574" watchObservedRunningTime="2026-11-01 22:07:53.949258731 +0000 UTC m=+1420.923161274" Nov 01 22:07:59.975055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:07:59.972254 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:08:04.711754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.711724 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:08:04.716935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.716918 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:04.722207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.722181 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e2a91d\"" Nov 01 22:08:04.722332 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.722212 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e2a91d-dockercfg-4wz6l\"" Nov 01 22:08:04.728163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.728142 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:08:04.760675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.760649 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:08:04.765344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.765323 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:04.776130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.776102 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:08:04.815832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.815804 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:04.816161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.816143 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jxg\" (UniqueName: \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:04.960064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.945625 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jxg\" (UniqueName: \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:04.960064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.945680 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:04.960064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.945744 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxghr\" (UniqueName: \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:04.960064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.945829 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:04.964091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.961057 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:05.002100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:04.993657 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jxg\" (UniqueName: \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\") pod \"payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:05.042053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.041321 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:05.114052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.108453 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:05.114052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.108539 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxghr\" (UniqueName: \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:05.119053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.116229 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:05.139420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.139290 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxghr\" (UniqueName: \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\") pod \"payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:05.389018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.388891 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:05.562084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.562054 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:08:05.562211 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:05.562115 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dec4122_ffdb_4733_8ebd_01be7394d85c.slice/crio-795ddfeccb9032a4f84116bd162e3dbc7741281734ef6018c4f6eb6850dc00e4 WatchSource:0}: Error finding container 795ddfeccb9032a4f84116bd162e3dbc7741281734ef6018c4f6eb6850dc00e4: Status 404 returned error can't find the container with id 795ddfeccb9032a4f84116bd162e3dbc7741281734ef6018c4f6eb6850dc00e4 Nov 01 22:08:05.700738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:05.700712 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:08:05.700844 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:05.700714 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3226fec_450e_4dce_8fae_56e2e9eaaba6.slice/crio-21826f4bee7237076db51d21ca8f637cd2182a027652fa2fa34738281669d9a5 WatchSource:0}: Error finding container 21826f4bee7237076db51d21ca8f637cd2182a027652fa2fa34738281669d9a5: Status 404 returned error can't find the container with id 21826f4bee7237076db51d21ca8f637cd2182a027652fa2fa34738281669d9a5 Nov 01 22:08:06.317011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.316971 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" event={"ID":"a3226fec-450e-4dce-8fae-56e2e9eaaba6","Type":"ContainerStarted","Data":"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020"} Nov 01 22:08:06.317011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.317048 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" event={"ID":"a3226fec-450e-4dce-8fae-56e2e9eaaba6","Type":"ContainerStarted","Data":"21826f4bee7237076db51d21ca8f637cd2182a027652fa2fa34738281669d9a5"} Nov 01 22:08:06.317566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.317083 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:06.318816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.318782 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" event={"ID":"3dec4122-ffdb-4733-8ebd-01be7394d85c","Type":"ContainerStarted","Data":"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56"} Nov 01 22:08:06.318948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.318835 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:06.318948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.318857 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" event={"ID":"3dec4122-ffdb-4733-8ebd-01be7394d85c","Type":"ContainerStarted","Data":"795ddfeccb9032a4f84116bd162e3dbc7741281734ef6018c4f6eb6850dc00e4"} Nov 01 22:08:06.340105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.340067 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" podStartSLOduration=2.340054849 podStartE2EDuration="2.340054849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:06.338960223 +0000 UTC m=+1433.312862763" watchObservedRunningTime="2026-11-01 22:08:06.340054849 +0000 UTC m=+1433.313957387" Nov 01 22:08:06.362966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:06.362923 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" podStartSLOduration=2.362911032 podStartE2EDuration="2.362911032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:06.360100152 +0000 UTC m=+1433.334002697" watchObservedRunningTime="2026-11-01 22:08:06.362911032 +0000 UTC m=+1433.336813570" Nov 01 22:08:11.261940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.261904 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:08:11.266861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.266841 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.280302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.280273 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:08:11.285612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.285591 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:08:11.285741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.285700 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.305691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.305649 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:08:11.408520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408480 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.408687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408602 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.408740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408691 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.408740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408732 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.408843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408827 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.408885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408857 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.408944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408888 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.408975 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.409126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409106 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.409182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409141 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409175 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.409346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409254 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409354 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409408 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4f2\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.409455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409430 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.409564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409456 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409554 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.409668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.409587 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk4fr\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511311 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511368 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4f2\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.511538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511497 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.511675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511607 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511644 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk4fr\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511697 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.511842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511766 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.511842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511835 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511863 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.511991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512072 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512101 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512160 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512193 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512407 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512439 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512470 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512505 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.512645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512568 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512627 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.512918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.512867 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.513334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513181 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.513334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513249 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.513551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513456 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.513551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513490 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.513551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513543 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.513734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.513662 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.514441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.514413 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.514822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.514788 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.515166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.515149 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.515231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.515192 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.524550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.524519 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk4fr\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.524642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.524519 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4f2\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.524853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.524832 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.525062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.525010 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.585911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.585882 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:11.625322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.625290 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:11.890907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.890881 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:08:11.890907 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:11.890891 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76a221b_6d77_4f50_8e74_59d8e0a1f862.slice/crio-42a1a8be3edeaf0f107c684c5926051506381a175e2ce8cab04b218df0d09a04 WatchSource:0}: Error finding container 42a1a8be3edeaf0f107c684c5926051506381a175e2ce8cab04b218df0d09a04: Status 404 returned error can't find the container with id 42a1a8be3edeaf0f107c684c5926051506381a175e2ce8cab04b218df0d09a04 Nov 01 22:08:11.893944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.893911 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:11.894080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.893975 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:11.894080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.894002 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:11.954265 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:11.954235 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a95cf6_2b84_4c29_980d_a0e0cc64557f.slice/crio-8bef07a1ab460bc853cdbc4c28ad75fef53e9f3a0d8cea5bb9c6a0d63950ae97 WatchSource:0}: Error finding container 8bef07a1ab460bc853cdbc4c28ad75fef53e9f3a0d8cea5bb9c6a0d63950ae97: Status 404 returned error can't find the container with id 8bef07a1ab460bc853cdbc4c28ad75fef53e9f3a0d8cea5bb9c6a0d63950ae97 Nov 01 22:08:11.954610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.954588 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:08:11.958068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.958019 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:11.958183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.958116 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:11.958183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:11.958161 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:12.572236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.572198 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" event={"ID":"c4a95cf6-2b84-4c29-980d-a0e0cc64557f","Type":"ContainerStarted","Data":"8bef07a1ab460bc853cdbc4c28ad75fef53e9f3a0d8cea5bb9c6a0d63950ae97"} Nov 01 22:08:12.574509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.574473 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" event={"ID":"d76a221b-6d77-4f50-8e74-59d8e0a1f862","Type":"ContainerStarted","Data":"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901"} Nov 01 22:08:12.574658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.574536 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" event={"ID":"d76a221b-6d77-4f50-8e74-59d8e0a1f862","Type":"ContainerStarted","Data":"42a1a8be3edeaf0f107c684c5926051506381a175e2ce8cab04b218df0d09a04"} Nov 01 22:08:12.586259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.586227 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:12.587770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.587736 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:12.587957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.587797 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:12.610604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:12.608202 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podStartSLOduration=1.608183413 podStartE2EDuration="1.608183413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:12.606864383 +0000 UTC m=+1439.580766925" watchObservedRunningTime="2026-11-01 22:08:12.608183413 +0000 UTC m=+1439.582085954" Nov 01 22:08:13.607975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:13.607388 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:13.607975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:13.607445 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:13.644065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:13.635095 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" event={"ID":"c4a95cf6-2b84-4c29-980d-a0e0cc64557f","Type":"ContainerStarted","Data":"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc"} Nov 01 22:08:14.596664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:14.594118 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:14.596664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:14.594197 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:14.634076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:14.630542 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:14.634076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:14.630697 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:14.634076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:14.630751 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:15.597544 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:15.597501 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:15.597838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:15.597804 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:15.635545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:15.635397 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:15.635545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:15.635521 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:16.587053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:16.586593 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:16.587053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:16.586702 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:16.627053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:16.626544 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:16.627053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:16.626621 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:17.465860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.465832 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:08:17.466424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.466407 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:08:17.496447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.489188 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podStartSLOduration=6.489169904 podStartE2EDuration="6.489169904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:13.677595282 +0000 UTC m=+1440.651497822" watchObservedRunningTime="2026-11-01 22:08:17.489169904 +0000 UTC m=+1444.463072444" Nov 01 22:08:17.634053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.611592 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:17.634354 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.634322 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:17.645645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.645613 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:17.645912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:17.645882 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:18.596055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:18.588215 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:18.596055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:18.588288 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:18.635253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:18.635216 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:18.635550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:18.635519 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:19.592056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:19.591439 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:19.592056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:19.591502 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:19.627498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:19.627460 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:19.628095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:19.628062 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:20.591525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:20.591485 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:08:20.591831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:20.591804 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:08:20.625890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:20.625851 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:08:20.626071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:20.625918 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:08:21.604315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.604284 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:21.606486 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.606464 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:21.607885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.607863 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:08:21.656802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.656774 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:21.658552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.658528 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:21.667327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:21.667300 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:08:22.353772 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:08:22.353739 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:08:31.200090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.200054 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:08:31.206602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.206578 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.216488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.216460 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:08:31.217818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.217783 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-487639-dockercfg-ftx4j\"" Nov 01 22:08:31.220405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.220380 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-487639\"" Nov 01 22:08:31.313231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.313194 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.313395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.313336 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmk72\" (UniqueName: \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.439059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.430328 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmk72\" (UniqueName: \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.439059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.430742 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.439059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.432057 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.486708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.463749 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmk72\" (UniqueName: \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\") pod \"payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:31.599481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:31.599443 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:32.189765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:32.189719 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:08:33.171020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:33.170983 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" event={"ID":"defccf48-c732-4e39-9c1b-8d2f467609c3","Type":"ContainerStarted","Data":"de9caf17fb14941c66e2b7e9a2ad801b2b58631d49fed5e6c82fb2b4fb6b2629"} Nov 01 22:08:33.171420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:33.171063 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" event={"ID":"defccf48-c732-4e39-9c1b-8d2f467609c3","Type":"ContainerStarted","Data":"467c336c6e50a414ac82bcd72b6fa356faac87fbd4c8f4b6022b186a9271ad08"} Nov 01 22:08:33.185154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:33.185101 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" podStartSLOduration=2.185083732 podStartE2EDuration="2.185083732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:33.180857565 +0000 UTC m=+1460.154760105" watchObservedRunningTime="2026-11-01 22:08:33.185083732 +0000 UTC m=+1460.158986275" Nov 01 22:08:34.168864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:34.168834 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:42.207487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.207456 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:08:42.220110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.219651 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.231549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.231517 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d8b5d-openshift-default-dockercfg-pxzwg\"" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236577 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236617 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236671 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236797 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236886 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.236913 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.237110 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.237182 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnrs\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.237426 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.238428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.237939 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:08:42.338838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.338799 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.338902 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.338927 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.338977 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339169 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339286 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339312 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339356 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339402 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339545 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339615 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnrs\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339627 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.339760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.339740 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.340447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.340417 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.342070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.341714 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.342070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.341744 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.351705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.351684 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.351914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.351894 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnrs\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.547979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.547952 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:42.984823 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:42.984794 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c6c593_66d7_4edb_ae89_0f4e1fd13365.slice/crio-37d993258c274a2b679bc5b710a6e6a83a16ebe56ae09be6da78cb444ef895c9 WatchSource:0}: Error finding container 37d993258c274a2b679bc5b710a6e6a83a16ebe56ae09be6da78cb444ef895c9: Status 404 returned error can't find the container with id 37d993258c274a2b679bc5b710a6e6a83a16ebe56ae09be6da78cb444ef895c9 Nov 01 22:08:42.985073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.985049 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:08:42.987890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.987862 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:42.987974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.987922 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:42.987974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:42.987953 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:08:43.308045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:43.307994 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" event={"ID":"66c6c593-66d7-4edb-ae89-0f4e1fd13365","Type":"ContainerStarted","Data":"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75"} Nov 01 22:08:43.308477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:43.308083 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" event={"ID":"66c6c593-66d7-4edb-ae89-0f4e1fd13365","Type":"ContainerStarted","Data":"37d993258c274a2b679bc5b710a6e6a83a16ebe56ae09be6da78cb444ef895c9"} Nov 01 22:08:43.334789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:43.332334 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podStartSLOduration=1.332317556 podStartE2EDuration="1.332317556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:43.330803593 +0000 UTC m=+1470.304706135" watchObservedRunningTime="2026-11-01 22:08:43.332317556 +0000 UTC m=+1470.306220095" Nov 01 22:08:43.548452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:43.548421 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:44.553548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:44.553518 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:44.553948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:44.553570 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:45.209704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:45.209650 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:08:45.549301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:45.549271 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:45.549474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:45.549321 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:46.548864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:46.548828 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:46.549279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:46.548895 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:47.554435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:47.554399 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:47.556825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:47.556789 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:48.548809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:48.548773 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:48.548990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:48.548832 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:49.548375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:49.548338 4395 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 22:08:49.548980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:49.548951 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 22:08:50.556615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:50.556576 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:50.559417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:50.559358 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:50.560234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:50.560210 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:08:51.047996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.047961 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:08:51.054778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.054747 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.068572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.068545 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-487639-serving-cert\"" Nov 01 22:08:51.074133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.074107 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:08:51.078125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.078095 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.078262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.078149 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhqs\" (UniqueName: \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.078356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.078323 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.078524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.078494 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.179805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.179772 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.179983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.179816 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhqs\" (UniqueName: \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.179983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.179961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.180156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.180138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.180740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.180713 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.182607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.182585 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.182731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.182710 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.244162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.240078 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhqs\" (UniqueName: \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\") pod \"maas-api-e2e-inf-a-487639-688874b489-png2n\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.387100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.386340 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:51.693524 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:08:51.693489 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a699c8_3443_4993_a7d6_74d8deb98576.slice/crio-2bb70cfdb744a48a63ac56ddba6e96e7f954caa7daf0c2548b3d868c16ae260e WatchSource:0}: Error finding container 2bb70cfdb744a48a63ac56ddba6e96e7f954caa7daf0c2548b3d868c16ae260e: Status 404 returned error can't find the container with id 2bb70cfdb744a48a63ac56ddba6e96e7f954caa7daf0c2548b3d868c16ae260e Nov 01 22:08:51.695018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:51.694991 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:08:52.334932 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:08:52.334890 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:08:52.548606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:52.548569 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" event={"ID":"c5a699c8-3443-4993-a7d6-74d8deb98576","Type":"ContainerStarted","Data":"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3"} Nov 01 22:08:52.548786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:52.548658 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:08:52.548786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:52.548683 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" event={"ID":"c5a699c8-3443-4993-a7d6-74d8deb98576","Type":"ContainerStarted","Data":"2bb70cfdb744a48a63ac56ddba6e96e7f954caa7daf0c2548b3d868c16ae260e"} Nov 01 22:08:52.571870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:52.571820 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" podStartSLOduration=1.171193324 podStartE2EDuration="1.571806203s" totalImagesPullingTime="400.612879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:52.567853227 +0000 UTC m=+1479.541755779" watchObservedRunningTime="2026-11-01 22:08:52.571806203 +0000 UTC m=+1479.545708763" Nov 01 22:08:58.564742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:08:58.564669 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:09:10.624701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:10.604404 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:09:15.628177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:15.628106 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" podUID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" containerName="payload-pre-processing" containerID="cri-o://86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39" gracePeriod=25 Nov 01 22:09:15.961823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:15.961797 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:09:16.053184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.053133 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\") pod \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " Nov 01 22:09:16.053386 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.053216 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\") pod \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\" (UID: \"af57c9eb-c623-4c3b-9ac9-73b08c696ddc\") " Nov 01 22:09:16.053990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.053963 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume" pod "af57c9eb-c623-4c3b-9ac9-73b08c696ddc" (UID: "af57c9eb-c623-4c3b-9ac9-73b08c696ddc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:09:16.054858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.054723 4395 generic.go:356] "Generic (PLEG): container finished" podID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" containerID="86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39" exitCode=0 Nov 01 22:09:16.054858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.054839 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" Nov 01 22:09:16.055064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.054937 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" event={"ID":"af57c9eb-c623-4c3b-9ac9-73b08c696ddc","Type":"ContainerDied","Data":"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39"} Nov 01 22:09:16.055199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.055021 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx" event={"ID":"af57c9eb-c623-4c3b-9ac9-73b08c696ddc","Type":"ContainerDied","Data":"35edbf19e2ab4e7848cd1a77b0bfe216a007ba3a94784d5c9eab1c96d4496726"} Nov 01 22:09:16.055268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.055233 4395 scope.go:122] "RemoveContainer" containerID="86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39" Nov 01 22:09:16.059160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.056551 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8" pod "af57c9eb-c623-4c3b-9ac9-73b08c696ddc" (UID: "af57c9eb-c623-4c3b-9ac9-73b08c696ddc"). InnerVolumeSpecName "kube-api-access-rqkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:09:16.078515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.078499 4395 scope.go:122] "RemoveContainer" containerID="86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39" Nov 01 22:09:16.078788 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:09:16.078764 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39\": container with ID starting with 86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39 not found: ID does not exist" containerID="86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39" Nov 01 22:09:16.078883 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.078793 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39"} err="failed to get container status \"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39\": rpc error: code = NotFound desc = could not find container \"86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39\": container with ID starting with 86b834da46844b732f76528f03a5f821bf4198922046fceecae3b7f78ce5bf39 not found: ID does not exist" Nov 01 22:09:16.175261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.160136 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqkn8\" (UniqueName: \"kubernetes.io/projected/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-kube-api-access-rqkn8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:09:16.175261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.160175 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af57c9eb-c623-4c3b-9ac9-73b08c696ddc-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:09:16.387389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.387360 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:09:16.395510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:16.395483 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0f9b7e-6f4456d5f-fchfx"] Nov 01 22:09:17.181252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.174322 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" path="/var/lib/kubelet/pods/af57c9eb-c623-4c3b-9ac9-73b08c696ddc/volumes" Nov 01 22:09:17.181252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.174707 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:09:17.181252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.175699 4395 cpu_manager.go:414] "containerMap: removing container" podUID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" containerName="payload-pre-processing" Nov 01 22:09:17.181252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.175714 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" containerName="payload-pre-processing" Nov 01 22:09:17.181252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.175845 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="af57c9eb-c623-4c3b-9ac9-73b08c696ddc" containerName="payload-pre-processing" Nov 01 22:09:17.181972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.181541 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:09:17.181972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.181640 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.198711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.194014 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dfdb11-serving-cert\"" Nov 01 22:09:17.301575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.301543 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.301730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.301591 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2sk\" (UniqueName: \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.301730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.301614 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.301811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.301783 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.405053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.404074 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.405053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.404478 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.405053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.404560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2sk\" (UniqueName: \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.405053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.404617 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.409053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.408923 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.446100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.445372 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.470053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.467791 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2sk\" (UniqueName: \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.473951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.473890 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\") pod \"maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.546117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.526383 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:17.923097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:17.923062 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:09:17.984379 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:09:17.984338 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74a83d8_861c_42fc_959e_f83751c4758f.slice/crio-e215ac2ba09cdf35e5a7ccaa3c372f42d23d3d57462bcdc167cdd43f440dbfb2 WatchSource:0}: Error finding container e215ac2ba09cdf35e5a7ccaa3c372f42d23d3d57462bcdc167cdd43f440dbfb2: Status 404 returned error can't find the container with id e215ac2ba09cdf35e5a7ccaa3c372f42d23d3d57462bcdc167cdd43f440dbfb2 Nov 01 22:09:18.074484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:18.074447 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" event={"ID":"d74a83d8-861c-42fc-959e-f83751c4758f","Type":"ContainerStarted","Data":"e215ac2ba09cdf35e5a7ccaa3c372f42d23d3d57462bcdc167cdd43f440dbfb2"} Nov 01 22:09:19.087975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:19.087932 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" event={"ID":"d74a83d8-861c-42fc-959e-f83751c4758f","Type":"ContainerStarted","Data":"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89"} Nov 01 22:09:19.088502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:19.088241 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:19.114323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:19.114268 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" podStartSLOduration=1.683905786 podStartE2EDuration="2.114254055s" totalImagesPullingTime="430.348269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:19.107759127 +0000 UTC m=+1506.081661668" watchObservedRunningTime="2026-11-01 22:09:19.114254055 +0000 UTC m=+1506.088156630" Nov 01 22:09:21.665203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.665169 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:09:21.670500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.670479 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:21.678613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.678583 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:09:21.681253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.681228 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3e300c-dockercfg-xfk26\"" Nov 01 22:09:21.681450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.681307 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3e300c\"" Nov 01 22:09:21.733395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.733365 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pklh\" (UniqueName: \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:21.733587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.733560 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:21.886053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.880668 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pklh\" (UniqueName: \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:21.886053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.880888 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:21.886053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:21.882048 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:22.005102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:22.001901 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pklh\" (UniqueName: \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\") pod \"payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:22.128110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:22.023262 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:22.343924 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:09:22.341733 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:09:22.544040 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:09:22.543987 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7dc798_11a9_4f2a_9314_7e7e4eca44c4.slice/crio-99018ac83d411f3f5593862f86b604701d173b36ba65d9feca904e798f20eddc WatchSource:0}: Error finding container 99018ac83d411f3f5593862f86b604701d173b36ba65d9feca904e798f20eddc: Status 404 returned error can't find the container with id 99018ac83d411f3f5593862f86b604701d173b36ba65d9feca904e798f20eddc Nov 01 22:09:22.544451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:22.544430 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:09:23.130685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:23.130640 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" event={"ID":"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4","Type":"ContainerStarted","Data":"4cff0fb6c98cc46f4f1ecfa39ff5bb696d580083432f295ac0975202e551fe03"} Nov 01 22:09:23.131141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:23.130748 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:23.131141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:23.130775 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" event={"ID":"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4","Type":"ContainerStarted","Data":"99018ac83d411f3f5593862f86b604701d173b36ba65d9feca904e798f20eddc"} Nov 01 22:09:23.149649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:23.149604 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" podStartSLOduration=2.149591054 podStartE2EDuration="2.149591054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:23.148552088 +0000 UTC m=+1510.122454641" watchObservedRunningTime="2026-11-01 22:09:23.149591054 +0000 UTC m=+1510.123493592" Nov 01 22:09:25.121715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:25.121682 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:09:29.630053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:29.629178 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.054131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.043261 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.074465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.073184 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.171289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.161826 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.173205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.173165 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.266487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.265571 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.396595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.395533 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.625052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.590222 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:30.625052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.592075 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:09:30.625052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.592460 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:09:30.673407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:30.673298 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.057575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.057353 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.090106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.059157 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:09:31.090106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.089236 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.258065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.254505 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.313192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.299882 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:09:31.313192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.302879 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:31.551392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.532409 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 22:09:31.611761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.609961 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:09:31.689406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.689366 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:31.774055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.748224 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Nov 01 22:09:31.794053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.758972 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:09:31.842053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.841350 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:09:31.842053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.841455 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:09:31.842053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.841498 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:09:31.842053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.841534 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:09:31.974364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:31.974273 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.064067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.059730 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.297325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.287151 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.334163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.307166 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:09:32.334163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.308099 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:09:32.388535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.388497 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c8vgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:09:32.388983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.388948 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.392499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.392470 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.557116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.549228 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.610915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.610870 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:32.690178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:32.690132 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.090053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.078611 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.360975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.360872 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:09:33.361750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.361721 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:09:33.432058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.406477 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 01 22:09:33.445057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.434277 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 22:09:33.632927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.627195 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:09:33.671455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.671230 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682623 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682689 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682721 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682752 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682787 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682816 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682857 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:09:33.698084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.682889 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:09:33.722004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.721978 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:09:33.732512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:33.732488 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:09:34.141314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:34.141283 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:09:41.856441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.856397 4395 generic.go:356] "Generic (PLEG): container finished" podID="07051168-24a0-49a9-a528-cc8098bf2692" containerID="f98bde87b5f0bf246ce431c7235a1fe5003a65cb1b215cf2346dc80abf9681aa" exitCode=0 Nov 01 22:09:41.856846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.856472 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerDied","Data":"f98bde87b5f0bf246ce431c7235a1fe5003a65cb1b215cf2346dc80abf9681aa"} Nov 01 22:09:41.856941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.856923 4395 scope.go:122] "RemoveContainer" containerID="f98bde87b5f0bf246ce431c7235a1fe5003a65cb1b215cf2346dc80abf9681aa" Nov 01 22:09:41.858934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.858873 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:09:41.859070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.858948 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:09:41.859070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:41.858993 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:09:42.866769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:42.866737 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerStarted","Data":"a13ebecf7d08cb166989007f59c8c6be79c90c39641386acf52cbe7d7ef8aab9"} Nov 01 22:09:43.068073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:43.067807 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:09:43.068073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:43.067957 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:43.068073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:43.068006 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:44.093456 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:44.093420 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:44.133055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:44.121753 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:45.069052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:45.064246 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:45.069052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:45.064325 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:45.898181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:45.898146 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:09:46.062080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:46.062052 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:46.062304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:46.062281 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:47.071401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:47.070872 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:47.071401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:47.070936 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:48.061607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:48.061574 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:48.061817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:48.061643 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:49.061774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:49.061732 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 01 22:09:49.062191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:49.061804 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 01 22:09:50.108764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:50.108734 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:09:50.114159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:09:50.114120 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:09:52.313264 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:09:52.313222 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:10:20.934279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:20.934240 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:10:20.947841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:20.947810 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:10:21.861054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:21.854088 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:10:22.010385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:21.999176 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:10:22.033340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.009169 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.121312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.104101 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-070e3d-dockercfg-r2cl6\"" Nov 01 22:10:22.121312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.109525 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-070e3d\"" Nov 01 22:10:22.169586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.169548 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g8r\" (UniqueName: \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.169835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.169813 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.271549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.271513 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g8r\" (UniqueName: \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.271784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.271556 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.272366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.272336 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.283984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.283957 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g8r\" (UniqueName: \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\") pod \"payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.312153 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:10:22.312110 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:10:22.473920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.473846 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:22.767636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:22.767608 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:10:22.767989 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:10:22.767963 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6396d252_a034_4f8e_b7c4_7d88278c2ef6.slice/crio-1a75927e70de6c852df8d936394757c35cddd9c6428c045d6a07566edf1e46d9 WatchSource:0}: Error finding container 1a75927e70de6c852df8d936394757c35cddd9c6428c045d6a07566edf1e46d9: Status 404 returned error can't find the container with id 1a75927e70de6c852df8d936394757c35cddd9c6428c045d6a07566edf1e46d9 Nov 01 22:10:23.752161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:23.751969 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" event={"ID":"6396d252-a034-4f8e-b7c4-7d88278c2ef6","Type":"ContainerStarted","Data":"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d"} Nov 01 22:10:23.752161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:23.752057 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" event={"ID":"6396d252-a034-4f8e-b7c4-7d88278c2ef6","Type":"ContainerStarted","Data":"1a75927e70de6c852df8d936394757c35cddd9c6428c045d6a07566edf1e46d9"} Nov 01 22:10:23.754265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:23.754190 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:25.934864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:25.934773 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" podUID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" containerName="payload-processing" containerID="cri-o://bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020" gracePeriod=25 Nov 01 22:10:25.948246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:25.948215 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" podUID="3dec4122-ffdb-4733-8ebd-01be7394d85c" containerName="payload-pre-processing" containerID="cri-o://18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56" gracePeriod=25 Nov 01 22:10:26.350861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.350836 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:10:26.379050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.373930 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" podStartSLOduration=5.373910748 podStartE2EDuration="5.373910748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:23.789088138 +0000 UTC m=+1570.762990677" watchObservedRunningTime="2026-11-01 22:10:26.373910748 +0000 UTC m=+1573.347813288" Nov 01 22:10:26.398194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.398167 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\") pod \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " Nov 01 22:10:26.398363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.398262 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\" (UniqueName: \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\") pod \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\" (UID: \"a3226fec-450e-4dce-8fae-56e2e9eaaba6\") " Nov 01 22:10:26.398554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.398530 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume" pod "a3226fec-450e-4dce-8fae-56e2e9eaaba6" (UID: "a3226fec-450e-4dce-8fae-56e2e9eaaba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:26.398971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.398950 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3226fec-450e-4dce-8fae-56e2e9eaaba6-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:26.404764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.404739 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr" pod "a3226fec-450e-4dce-8fae-56e2e9eaaba6" (UID: "a3226fec-450e-4dce-8fae-56e2e9eaaba6"). InnerVolumeSpecName "kube-api-access-xxghr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:26.431722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.431693 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:10:26.499685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.499656 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\") pod \"3dec4122-ffdb-4733-8ebd-01be7394d85c\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " Nov 01 22:10:26.500001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.499986 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\" (UniqueName: \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\") pod \"3dec4122-ffdb-4733-8ebd-01be7394d85c\" (UID: \"3dec4122-ffdb-4733-8ebd-01be7394d85c\") " Nov 01 22:10:26.500154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.500093 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume" pod "3dec4122-ffdb-4733-8ebd-01be7394d85c" (UID: "3dec4122-ffdb-4733-8ebd-01be7394d85c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:10:26.500939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.500904 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dec4122-ffdb-4733-8ebd-01be7394d85c-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:26.500939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.500935 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxghr\" (UniqueName: \"kubernetes.io/projected/a3226fec-450e-4dce-8fae-56e2e9eaaba6-kube-api-access-xxghr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:26.502412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.502383 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg" pod "3dec4122-ffdb-4733-8ebd-01be7394d85c" (UID: "3dec4122-ffdb-4733-8ebd-01be7394d85c"). InnerVolumeSpecName "kube-api-access-n6jxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:10:26.602691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.602626 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jxg\" (UniqueName: \"kubernetes.io/projected/3dec4122-ffdb-4733-8ebd-01be7394d85c-kube-api-access-n6jxg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:10:26.807978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.807945 4395 generic.go:356] "Generic (PLEG): container finished" podID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" containerID="bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020" exitCode=0 Nov 01 22:10:26.808203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.807990 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" event={"ID":"a3226fec-450e-4dce-8fae-56e2e9eaaba6","Type":"ContainerDied","Data":"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020"} Nov 01 22:10:26.808203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.808068 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" event={"ID":"a3226fec-450e-4dce-8fae-56e2e9eaaba6","Type":"ContainerDied","Data":"21826f4bee7237076db51d21ca8f637cd2182a027652fa2fa34738281669d9a5"} Nov 01 22:10:26.808203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.807997 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc" Nov 01 22:10:26.808203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.808094 4395 scope.go:122] "RemoveContainer" containerID="bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020" Nov 01 22:10:26.809901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.809878 4395 generic.go:356] "Generic (PLEG): container finished" podID="3dec4122-ffdb-4733-8ebd-01be7394d85c" containerID="18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56" exitCode=0 Nov 01 22:10:26.810049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.809922 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" event={"ID":"3dec4122-ffdb-4733-8ebd-01be7394d85c","Type":"ContainerDied","Data":"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56"} Nov 01 22:10:26.810049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.809961 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" event={"ID":"3dec4122-ffdb-4733-8ebd-01be7394d85c","Type":"ContainerDied","Data":"795ddfeccb9032a4f84116bd162e3dbc7741281734ef6018c4f6eb6850dc00e4"} Nov 01 22:10:26.810049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.809925 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv" Nov 01 22:10:26.820985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.820968 4395 scope.go:122] "RemoveContainer" containerID="bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020" Nov 01 22:10:26.821268 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:10:26.821250 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020\": container with ID starting with bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020 not found: ID does not exist" containerID="bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020" Nov 01 22:10:26.821346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.821279 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020"} err="failed to get container status \"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020\": rpc error: code = NotFound desc = could not find container \"bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020\": container with ID starting with bfcfa63c0fd82abb9392d5a59717f2a1c04142d804169df72f3dc67f2b6d6020 not found: ID does not exist" Nov 01 22:10:26.821346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.821300 4395 scope.go:122] "RemoveContainer" containerID="18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56" Nov 01 22:10:26.832209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.832189 4395 scope.go:122] "RemoveContainer" containerID="18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56" Nov 01 22:10:26.832511 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:10:26.832496 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56\": container with ID starting with 18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56 not found: ID does not exist" containerID="18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56" Nov 01 22:10:26.832573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.832517 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56"} err="failed to get container status \"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56\": rpc error: code = NotFound desc = could not find container \"18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56\": container with ID starting with 18ba2d10c955857a666489ab5426d08d9dc7f312ccbf8215d0f67da10ba7ee56 not found: ID does not exist" Nov 01 22:10:26.836146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.836128 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:10:26.842165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.842147 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e2a91d-788c9d6d68-fb2lc"] Nov 01 22:10:26.864713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.864645 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:10:26.864713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:26.864696 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2a91d-544d8bc56c-m9tgv"] Nov 01 22:10:27.181816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:27.181734 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dec4122-ffdb-4733-8ebd-01be7394d85c" path="/var/lib/kubelet/pods/3dec4122-ffdb-4733-8ebd-01be7394d85c/volumes" Nov 01 22:10:27.182792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:27.182768 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" path="/var/lib/kubelet/pods/a3226fec-450e-4dce-8fae-56e2e9eaaba6/volumes" Nov 01 22:10:35.822058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:10:35.816913 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:10:52.304458 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:10:52.304418 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:11:22.304820 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:11:22.304753 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:11:52.304710 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:11:52.304671 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:12:22.372980 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:12:22.372938 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:12:28.418076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.418020 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:12:28.419486 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419460 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" containerName="payload-processing" Nov 01 22:12:28.419601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419488 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" containerName="payload-processing" Nov 01 22:12:28.419601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419519 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3dec4122-ffdb-4733-8ebd-01be7394d85c" containerName="payload-pre-processing" Nov 01 22:12:28.419601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419528 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dec4122-ffdb-4733-8ebd-01be7394d85c" containerName="payload-pre-processing" Nov 01 22:12:28.419771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419710 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3226fec-450e-4dce-8fae-56e2e9eaaba6" containerName="payload-processing" Nov 01 22:12:28.419771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.419729 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dec4122-ffdb-4733-8ebd-01be7394d85c" containerName="payload-pre-processing" Nov 01 22:12:28.436594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.436568 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.442227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.442205 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-070e3d\"/\"default-dockercfg-7bbzr\"" Nov 01 22:12:28.442414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.442243 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-070e3d\"/\"openshift-service-ca.crt\"" Nov 01 22:12:28.442511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.442246 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-070e3d\"/\"kube-root-ca.crt\"" Nov 01 22:12:28.442511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.442338 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-070e3d\"/\"test-model-070e3d-kserve-self-signed-certs\"" Nov 01 22:12:28.449542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.449188 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.615986 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5px2\" (UniqueName: \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.616078 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.616292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.616572 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.616616 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.622062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.616741 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.725987 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.726213 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5px2\" (UniqueName: \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.726267 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.726399 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.726629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.726658 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.727226 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.728116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.727892 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.748063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.747269 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.751723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.751691 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.753723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.753672 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:28.777996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:28.777943 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5px2\" (UniqueName: \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\") pod \"test-model-070e3d-kserve-77fcddbf7c-gjqh8\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:29.058284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:29.058255 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:29.405846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:29.405817 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:12:30.038536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:30.038500 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" event={"ID":"4d30a263-029d-465f-ba01-5de67184fa12","Type":"ContainerStarted","Data":"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828"} Nov 01 22:12:30.038933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:30.038545 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" event={"ID":"4d30a263-029d-465f-ba01-5de67184fa12","Type":"ContainerStarted","Data":"243195cd8a474af14de8d3a6fefb0669989e84159c76f6946f9e3d344396ac14"} Nov 01 22:12:30.076055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:30.075981 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:30.083889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:30.077144 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" podStartSLOduration=2.077127502 podStartE2EDuration="2.077127502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:12:30.057282889 +0000 UTC m=+1697.031185439" watchObservedRunningTime="2026-11-01 22:12:30.077127502 +0000 UTC m=+1697.051030044" Nov 01 22:12:39.059380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:39.059341 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:39.073658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:39.073626 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:39.223558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:12:39.223524 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:12:52.429864 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:12:52.429778 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:13:22.308717 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:13:22.308669 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:13:33.870058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:33.869990 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:13:38.870861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:38.870823 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" containerID="cri-o://e4a11825d8a17c3e72132c95157a0d464c14d660ffbb7bfcec2fc0955b3d2d79" gracePeriod=25 Nov 01 22:13:42.016251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:42.016210 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:13:47.670439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:47.670399 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:13:47.670956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:47.670662 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" podUID="673906a5-1463-4b6e-bdf8-6579af0f3175" containerName="maas-api" containerID="cri-o://a6e5ec69ff613e4046194df9aaa441ff18fc1f86fa4bdd277331bfc9098f9073" gracePeriod=30 Nov 01 22:13:48.468095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.468059 4395 generic.go:356] "Generic (PLEG): container finished" podID="673906a5-1463-4b6e-bdf8-6579af0f3175" containerID="a6e5ec69ff613e4046194df9aaa441ff18fc1f86fa4bdd277331bfc9098f9073" exitCode=0 Nov 01 22:13:48.468599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.468570 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" event={"ID":"673906a5-1463-4b6e-bdf8-6579af0f3175","Type":"ContainerDied","Data":"a6e5ec69ff613e4046194df9aaa441ff18fc1f86fa4bdd277331bfc9098f9073"} Nov 01 22:13:48.588514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.588481 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:13:48.662820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.662718 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:13:48.666158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.665760 4395 cpu_manager.go:414] "containerMap: removing container" podUID="673906a5-1463-4b6e-bdf8-6579af0f3175" containerName="maas-api" Nov 01 22:13:48.666371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.666349 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673906a5-1463-4b6e-bdf8-6579af0f3175" containerName="maas-api" Nov 01 22:13:48.666801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.666780 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="673906a5-1463-4b6e-bdf8-6579af0f3175" containerName="maas-api" Nov 01 22:13:48.696693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.696655 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\") pod \"673906a5-1463-4b6e-bdf8-6579af0f3175\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " Nov 01 22:13:48.706911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.706878 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\") pod \"673906a5-1463-4b6e-bdf8-6579af0f3175\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " Nov 01 22:13:48.707614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.707590 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\") pod \"673906a5-1463-4b6e-bdf8-6579af0f3175\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " Nov 01 22:13:48.708280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.708262 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\" (UniqueName: \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\") pod \"673906a5-1463-4b6e-bdf8-6579af0f3175\" (UID: \"673906a5-1463-4b6e-bdf8-6579af0f3175\") " Nov 01 22:13:48.751118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.697628 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.751118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.697529 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:13:48.771185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.699652 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca" pod "673906a5-1463-4b6e-bdf8-6579af0f3175" (UID: "673906a5-1463-4b6e-bdf8-6579af0f3175"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:48.771185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.769176 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw" pod "673906a5-1463-4b6e-bdf8-6579af0f3175" (UID: "673906a5-1463-4b6e-bdf8-6579af0f3175"). InnerVolumeSpecName "kube-api-access-p8wfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:48.775701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.771609 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-070e3d-openshift-default-dockercfg-qwkpd\"" Nov 01 22:13:48.775934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.775908 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls" pod "673906a5-1463-4b6e-bdf8-6579af0f3175" (UID: "673906a5-1463-4b6e-bdf8-6579af0f3175"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:48.783774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.783740 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs" pod "673906a5-1463-4b6e-bdf8-6579af0f3175" (UID: "673906a5-1463-4b6e-bdf8-6579af0f3175"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:13:48.866801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.866761 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.866921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.866811 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.866921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.866863 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867012 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867164 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867202 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867302 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2dx\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867349 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.867622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867606 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673906a5-1463-4b6e-bdf8-6579af0f3175-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:48.867658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867628 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:48.867658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867637 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673906a5-1463-4b6e-bdf8-6579af0f3175-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:48.867725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.867648 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wfw\" (UniqueName: \"kubernetes.io/projected/673906a5-1463-4b6e-bdf8-6579af0f3175-kube-api-access-p8wfw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:48.969714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969632 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969676 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969700 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969784 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969858 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969888 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969919 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.969990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.969952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2dx\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.970281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.970011 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.970281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.970009 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.970281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.970115 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.970554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.970531 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.970789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.970737 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.971152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.971125 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.972465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.972446 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.972585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.972570 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.982306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.982288 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:48.982534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:48.982512 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2dx\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\") pod \"e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:49.116616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.116581 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:49.346542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.346515 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 22:13:49.346752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.346731 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" containerID="cri-o://663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac" gracePeriod=30 Nov 01 22:13:49.352894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.352864 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 22:13:49.353102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.353076 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" containerID="cri-o://e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd" gracePeriod=30 Nov 01 22:13:49.432253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.432220 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:13:49.436693 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:13:49.436662 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639c72f_460d_42d2_a9c6_e5cc6b767400.slice/crio-55d833fc5f27dfa7e55d6d6f2eb5b84828e670964107b6c9278101746437b6ed WatchSource:0}: Error finding container 55d833fc5f27dfa7e55d6d6f2eb5b84828e670964107b6c9278101746437b6ed: Status 404 returned error can't find the container with id 55d833fc5f27dfa7e55d6d6f2eb5b84828e670964107b6c9278101746437b6ed Nov 01 22:13:49.439843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.439804 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:49.439953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.439888 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:49.439953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.439933 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:49.440340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.440316 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 22:13:49.440899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.440532 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" containerID="cri-o://b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278" gracePeriod=30 Nov 01 22:13:49.499344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.499303 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" event={"ID":"7639c72f-460d-42d2-a9c6-e5cc6b767400","Type":"ContainerStarted","Data":"55d833fc5f27dfa7e55d6d6f2eb5b84828e670964107b6c9278101746437b6ed"} Nov 01 22:13:49.501916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.501890 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" event={"ID":"673906a5-1463-4b6e-bdf8-6579af0f3175","Type":"ContainerDied","Data":"97d73b1ec76545f90d4682831a1fc240676ac11d23455f87aaa1539a375bc33b"} Nov 01 22:13:49.502104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.501938 4395 scope.go:122] "RemoveContainer" containerID="a6e5ec69ff613e4046194df9aaa441ff18fc1f86fa4bdd277331bfc9098f9073" Nov 01 22:13:49.502186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.502116 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch" Nov 01 22:13:49.543422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.543383 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:13:49.551281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:49.551257 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edd32a-6574b588f7-bgxch"] Nov 01 22:13:50.512143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:50.512108 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" event={"ID":"7639c72f-460d-42d2-a9c6-e5cc6b767400","Type":"ContainerStarted","Data":"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674"} Nov 01 22:13:50.541325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:50.541266 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podStartSLOduration=19.541247447 podStartE2EDuration="19.541247447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:50.537650685 +0000 UTC m=+1777.511553225" watchObservedRunningTime="2026-11-01 22:13:50.541247447 +0000 UTC m=+1777.515150026" Nov 01 22:13:51.117339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:51.117307 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:51.118761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:51.118733 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:51.118900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:51.118783 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:51.212809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:51.212182 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673906a5-1463-4b6e-bdf8-6579af0f3175" path="/var/lib/kubelet/pods/673906a5-1463-4b6e-bdf8-6579af0f3175/volumes" Nov 01 22:13:52.057872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:52.056463 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:13:52.117853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:52.117822 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:52.118002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:52.117882 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:52.331435 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:13:52.331350 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:13:53.117886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:53.117854 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:53.118449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:53.117910 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:54.124053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:54.117547 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:54.124053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:54.117632 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:55.116065 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:13:55.092357 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8a5c4_19bf_41b0_a20c_b773c93a02dc.slice/crio-conmon-e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8a5c4_19bf_41b0_a20c_b773c93a02dc.slice/crio-e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376beac4_30fa_443f_8345_5d9a0d85ac94.slice/crio-conmon-b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb99ef7_62ca_4357_bec2_09ef732562fb.slice/crio-663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb99ef7_62ca_4357_bec2_09ef732562fb.slice/crio-conmon-663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:13:55.154061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.116991 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:55.154061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.117098 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:55.541754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.541725 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568230 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568335 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568377 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568412 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568453 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568513 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568686 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568884 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.568914 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\") pod \"2fb99ef7-62ca-4357-bec2-09ef732562fb\" (UID: \"2fb99ef7-62ca-4357-bec2-09ef732562fb\") " Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569207 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569299 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569522 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569727 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569857 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569878 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569895 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb99ef7-62ca-4357-bec2-09ef732562fb-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.569909 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.570154 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:55.578832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.573458 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:55.612929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.591506 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.638237 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.638450 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7" pod "2fb99ef7-62ca-4357-bec2-09ef732562fb" (UID: "2fb99ef7-62ca-4357-bec2-09ef732562fb"). InnerVolumeSpecName "kube-api-access-frqt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.646991 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.648547 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.648565 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" Nov 01 22:13:55.651306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.648945 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerName="istio-proxy" Nov 01 22:13:55.662570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.661683 4395 generic.go:356] "Generic (PLEG): container finished" podID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerID="e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd" exitCode=0 Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.664145 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" event={"ID":"eda8a5c4-19bf-41b0-a20c-b773c93a02dc","Type":"ContainerDied","Data":"e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd"} Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.664756 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.664256 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.668909 4395 generic.go:356] "Generic (PLEG): container finished" podID="2fb99ef7-62ca-4357-bec2-09ef732562fb" containerID="663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac" exitCode=0 Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.669085 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.669963 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3e300c-openshift-default-dockercfg-lpvl5\"" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670815 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670869 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670892 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670913 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9q6\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670943 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.670966 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671014 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671068 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671084 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671221 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" event={"ID":"2fb99ef7-62ca-4357-bec2-09ef732562fb","Type":"ContainerDied","Data":"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac"} Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671247 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671264 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn" event={"ID":"2fb99ef7-62ca-4357-bec2-09ef732562fb","Type":"ContainerDied","Data":"2446f64271e864509e4f29d09195bf35e27cd5436bf709046c098221637c8525"} Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671267 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671294 4395 scope.go:122] "RemoveContainer" containerID="663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671300 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671310 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.740405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.671320 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqt7\" (UniqueName: \"kubernetes.io/projected/2fb99ef7-62ca-4357-bec2-09ef732562fb-kube-api-access-frqt7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.777105 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.777485 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.777525 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9q6\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.782601 4395 generic.go:356] "Generic (PLEG): container finished" podID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerID="b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278" exitCode=0 Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.782692 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" event={"ID":"376beac4-30fa-443f-8345-5d9a0d85ac94","Type":"ContainerDied","Data":"b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278"} Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.783249 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.788351 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.825767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.800257 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:55.928160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.927463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.987294 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.987498 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.987539 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.987561 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.988079 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.988565 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.989338 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.030828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:55.989579 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.110308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.107435 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 22:13:56.129957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.129931 4395 scope.go:122] "RemoveContainer" containerID="663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac" Nov 01 22:13:56.134591 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.134131 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 22:13:56.136969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.136885 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 22:13:56.143521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.143490 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:56.143642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.143544 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:56.144156 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:13:56.144122 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac\": container with ID starting with 663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac not found: ID does not exist" containerID="663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac" Nov 01 22:13:56.144262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.144167 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac"} err="failed to get container status \"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac\": rpc error: code = NotFound desc = could not find container \"663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac\": container with ID starting with 663873565ca3ec1424f27f8a79a6e6a7446d1d97c735eb425c27228f9de63fac not found: ID does not exist" Nov 01 22:13:56.149117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.149066 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9q6\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.150833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.150809 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-n4pvn"] Nov 01 22:13:56.151396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.151371 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.151981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.151863 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312352 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312426 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312515 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312580 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312663 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.314068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.312696 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.325365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.325325 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.332539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.332496 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:56.333155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.332744 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.333361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.333335 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.334230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.334192 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.334751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.334732 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.342097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.333480 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.342225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.334510 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.342339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.342314 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.342463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.342432 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.342897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.342879 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.359467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.359429 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.388054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.345387 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.394131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.394102 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.394389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.394367 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.394542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.394524 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.394656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.394642 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\") pod \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\" (UID: \"eda8a5c4-19bf-41b0-a20c-b773c93a02dc\") " Nov 01 22:13:56.394795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.394780 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\") pod \"376beac4-30fa-443f-8345-5d9a0d85ac94\" (UID: \"376beac4-30fa-443f-8345-5d9a0d85ac94\") " Nov 01 22:13:56.396095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396074 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.396314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396266 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.396483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396467 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.396594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396582 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.396683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396671 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.396766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.396755 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.402098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.401013 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402545 4395 cpu_manager.go:414] "containerMap: removing container" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402568 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402587 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402595 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402844 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" Nov 01 22:13:56.405234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.402859 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" containerName="istio-proxy" Nov 01 22:13:56.405593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.405527 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "kube-api-access-dlr77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:56.405790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.405729 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.408081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.406668 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:56.431483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.431456 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:56.434157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.434134 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:13:56.434418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.434400 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:56.440300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.438419 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:13:56.451074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.447541 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:13:56.455569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.455542 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.456099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.456014 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:56.456547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.456494 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.456648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.456556 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:13:56.457510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.457478 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:13:56.462202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.462167 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "kube-api-access-xdm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:56.466493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.465403 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:13:56.466493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.465610 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" containerID="cri-o://81a0ca7b2b7806acb95f5b36029f5995dc65eead6784739e64a911180ea27934" gracePeriod=30 Nov 01 22:13:56.476846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.472562 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.532097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.521900 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:13:56.595222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.593675 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token" pod "376beac4-30fa-443f-8345-5d9a0d85ac94" (UID: "376beac4-30fa-443f-8345-5d9a0d85ac94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:56.595415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.593775 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token" pod "eda8a5c4-19bf-41b0-a20c-b773c93a02dc" (UID: "eda8a5c4-19bf-41b0-a20c-b773c93a02dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:13:56.607059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.601752 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:13:56.607059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.601829 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:13:56.775635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.731094 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:13:56.775635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.731632 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" containerID="cri-o://df143d6d6101e44c98f46c8e54f31fe502de1ed66ecbf2c623a8521f3b87f765" gracePeriod=30 Nov 01 22:13:56.880161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.880073 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdm5q\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-kube-api-access-xdm5q\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.880161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.880105 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.894482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.880130 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.902103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.894718 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.902407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.902353 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.908237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.908178 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376beac4-30fa-443f-8345-5d9a0d85ac94-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.908523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.908466 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.908523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.908495 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlr77\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-kube-api-access-dlr77\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.919497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.919453 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.919797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.919760 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376beac4-30fa-443f-8345-5d9a0d85ac94-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.922970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.922940 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:56.923623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:56.923602 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8a5c4-19bf-41b0-a20c-b773c93a02dc-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:13:57.006889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.006859 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" Nov 01 22:13:57.019523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.006906 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5" event={"ID":"eda8a5c4-19bf-41b0-a20c-b773c93a02dc","Type":"ContainerDied","Data":"42ab6a9f1d77cf85de44a171c90c0c56bcf227c9371c8568be4fbfa591d2133b"} Nov 01 22:13:57.019523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.014136 4395 scope.go:122] "RemoveContainer" containerID="e4e7f25672e7449231daae8f8b286c4053ac654a864854abe6ed33d91beb51bd" Nov 01 22:13:57.019523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.018702 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" event={"ID":"376beac4-30fa-443f-8345-5d9a0d85ac94","Type":"ContainerDied","Data":"bf5109f9cdf144308d5e2274e29e98e63b66c022dadd592902bf25e72c9b9cd7"} Nov 01 22:13:57.019523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.018883 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n" Nov 01 22:13:57.026376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026188 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.026376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026282 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026481 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026549 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026621 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026659 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026714 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026743 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.026853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.026789 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.027260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027074 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.027260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027126 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.027260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027179 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.027260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027227 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.040390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027525 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mvv\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.040390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027675 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.040390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027764 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.040390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027796 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.040390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.027841 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs55w\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.042230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.042207 4395 scope.go:122] "RemoveContainer" containerID="b166821859935d9910ee5158ceedd845f1754ec56c1a1f2477a46dbbc8c62278" Nov 01 22:13:57.073608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.073572 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 22:13:57.088985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.088948 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-sxkn5"] Nov 01 22:13:57.103105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.102688 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 22:13:57.116663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.114626 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-8z97n"] Nov 01 22:13:57.118396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.117965 4395 patch_prober.go:29] interesting pod/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:13:57.118396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.118049 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:13:57.128950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.128926 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.128975 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129014 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129105 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129141 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129194 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129222 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129259 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129302 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129330 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129390 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129437 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129495 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mvv\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.129880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129642 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.129880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.129871 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.130184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130124 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.130184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130173 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.130305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130223 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs55w\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.130305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130236 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.131766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130376 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.131766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.130720 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.131766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.131161 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.139257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.139229 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.139786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.139762 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.143208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.143182 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.145164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.145142 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.145567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.145548 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.146799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.146345 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mvv\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.148896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.148867 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.151903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.151855 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.152520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.152269 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.152520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.152476 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.153114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.153075 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs55w\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.153374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.153344 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.154843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.154820 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.177907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.177878 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb99ef7-62ca-4357-bec2-09ef732562fb" path="/var/lib/kubelet/pods/2fb99ef7-62ca-4357-bec2-09ef732562fb/volumes" Nov 01 22:13:57.179125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.179097 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376beac4-30fa-443f-8345-5d9a0d85ac94" path="/var/lib/kubelet/pods/376beac4-30fa-443f-8345-5d9a0d85ac94/volumes" Nov 01 22:13:57.179846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.179806 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda8a5c4-19bf-41b0-a20c-b773c93a02dc" path="/var/lib/kubelet/pods/eda8a5c4-19bf-41b0-a20c-b773c93a02dc/volumes" Nov 01 22:13:57.256933 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:13:57.256896 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbf8bf1_b905_4706_a6be_f23d5eb3bc62.slice/crio-4333b6ead0a0c04787c953ead9963cf1354e573304506a54c11133f628bbd1bb WatchSource:0}: Error finding container 4333b6ead0a0c04787c953ead9963cf1354e573304506a54c11133f628bbd1bb: Status 404 returned error can't find the container with id 4333b6ead0a0c04787c953ead9963cf1354e573304506a54c11133f628bbd1bb Nov 01 22:13:57.257089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.257062 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:13:57.260417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.260385 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.260547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.260468 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.260547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.260519 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.303662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.303636 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:57.410209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.410181 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:57.822314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.822284 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:13:57.824816 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:13:57.824776 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adb9d11_b88e_487d_a352_1a0bd06e100d.slice/crio-0647cd0341937b04b14700f5373376ad284f33ed5b909c19ad87c13ee62289de WatchSource:0}: Error finding container 0647cd0341937b04b14700f5373376ad284f33ed5b909c19ad87c13ee62289de: Status 404 returned error can't find the container with id 0647cd0341937b04b14700f5373376ad284f33ed5b909c19ad87c13ee62289de Nov 01 22:13:57.828940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.828905 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.829084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.828989 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.829084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.829073 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.940406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.940216 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:13:57.943314 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:13:57.943282 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edd1cae_0185_4d2c_88aa_d823651c7a45.slice/crio-d5989aa14fbc7f615cb97a794d63cb5b9f63ace075649ebc029fd9cb58304bbe WatchSource:0}: Error finding container d5989aa14fbc7f615cb97a794d63cb5b9f63ace075649ebc029fd9cb58304bbe: Status 404 returned error can't find the container with id d5989aa14fbc7f615cb97a794d63cb5b9f63ace075649ebc029fd9cb58304bbe Nov 01 22:13:57.948392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.948298 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.948503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.948433 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:57.948746 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:57.948719 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:13:58.038384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.038349 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" event={"ID":"1adb9d11-b88e-487d-a352-1a0bd06e100d","Type":"ContainerStarted","Data":"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6"} Nov 01 22:13:58.038637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.038618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" event={"ID":"1adb9d11-b88e-487d-a352-1a0bd06e100d","Type":"ContainerStarted","Data":"0647cd0341937b04b14700f5373376ad284f33ed5b909c19ad87c13ee62289de"} Nov 01 22:13:58.041871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.041841 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" event={"ID":"7edd1cae-0185-4d2c-88aa-d823651c7a45","Type":"ContainerStarted","Data":"d5989aa14fbc7f615cb97a794d63cb5b9f63ace075649ebc029fd9cb58304bbe"} Nov 01 22:13:58.047830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.047801 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" event={"ID":"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62","Type":"ContainerStarted","Data":"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115"} Nov 01 22:13:58.047957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.047851 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" event={"ID":"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62","Type":"ContainerStarted","Data":"4333b6ead0a0c04787c953ead9963cf1354e573304506a54c11133f628bbd1bb"} Nov 01 22:13:58.079562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.079499 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podStartSLOduration=72.079477132 podStartE2EDuration="1m12.079477132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:58.066377553 +0000 UTC m=+1785.040280092" watchObservedRunningTime="2026-11-01 22:13:58.079477132 +0000 UTC m=+1785.053379672" Nov 01 22:13:58.106097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.106000 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podStartSLOduration=87.105980566 podStartE2EDuration="1m27.105980566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:58.100677346 +0000 UTC m=+1785.074579889" watchObservedRunningTime="2026-11-01 22:13:58.105980566 +0000 UTC m=+1785.079883107" Nov 01 22:13:58.127214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.125312 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:58.127214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.127046 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:58.135378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.133360 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:13:58.303936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.303902 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:13:58.305294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.305259 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:13:58.305446 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.305317 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:13:58.448773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.448743 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:13:58.449172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.448780 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:13:58.449172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:58.449163 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:13:59.067081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.062631 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" event={"ID":"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe","Type":"ContainerDied","Data":"e4a11825d8a17c3e72132c95157a0d464c14d660ffbb7bfcec2fc0955b3d2d79"} Nov 01 22:13:59.067081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.062588 4395 generic.go:356] "Generic (PLEG): container finished" podID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerID="e4a11825d8a17c3e72132c95157a0d464c14d660ffbb7bfcec2fc0955b3d2d79" exitCode=0 Nov 01 22:13:59.067081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.065558 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" event={"ID":"7edd1cae-0185-4d2c-88aa-d823651c7a45","Type":"ContainerStarted","Data":"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a"} Nov 01 22:13:59.153504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.151781 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podStartSLOduration=88.151761654 podStartE2EDuration="1m28.151761654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:59.093555047 +0000 UTC m=+1786.067457588" watchObservedRunningTime="2026-11-01 22:13:59.151761654 +0000 UTC m=+1786.125664195" Nov 01 22:13:59.519383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.498882 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:13:59.668281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.665506 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:13:59.668281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.557713 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:13:59.668281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.557274 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:13:59.668281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.666194 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:13:59.778271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.763658 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:13:59.778271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.763721 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:13:59.888288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:13:59.888253 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:14:00.133982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.133883 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\") pod \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " Nov 01 22:14:00.257423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.257381 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\" (UniqueName: \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\") pod \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\" (UID: \"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe\") " Nov 01 22:14:00.305092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.171793 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume" pod "ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" (UID: "ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:00.305092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.301257 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.369505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.369457 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.369831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.369802 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:00.372114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.369833 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:00.425805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.418966 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.426638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.426606 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:00.453667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.453623 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" event={"ID":"ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe","Type":"ContainerDied","Data":"6634d575abbd8dd9519149cf2b322a9f5aa2b829ba876f6c6ca31732a6da0882"} Nov 01 22:14:00.453874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.453718 4395 scope.go:122] "RemoveContainer" containerID="e4a11825d8a17c3e72132c95157a0d464c14d660ffbb7bfcec2fc0955b3d2d79" Nov 01 22:14:00.454959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.454858 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p" Nov 01 22:14:00.464133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.463267 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:14:00.466117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.465428 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" Nov 01 22:14:00.466117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.465447 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" Nov 01 22:14:00.466117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.465886 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" containerName="payload-pre-processing" Nov 01 22:14:00.487300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.487222 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:14:00.487570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.487540 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:14:00.596135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.593523 4395 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.673055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.653453 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.755871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.755838 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl" pod "ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" (UID: "ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe"). InnerVolumeSpecName "kube-api-access-z8fnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:00.780229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.780189 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:14:00.781609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.781449 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.802879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.796381 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:00.827316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.827269 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907092 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907263 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907361 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907388 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907473 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907635 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxt6\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907669 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907727 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.907772 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:00.910052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:00.908020 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fnl\" (UniqueName: \"kubernetes.io/projected/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe-kube-api-access-z8fnl\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:01.027373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.027329 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.049173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.049134 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.049482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.049461 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.049765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.049743 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxt6\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.049935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.049919 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.050106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.050089 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.050260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.050245 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.050659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.050636 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.050984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.050961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.051610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.051587 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.051745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.029533 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.052849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.052821 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.055160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.054687 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.057661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.057636 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.062406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.062378 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.066314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.066251 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxt6\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.067216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.067190 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.067886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.067864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.273055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.272308 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:01.314329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.295646 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:14:01.314329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.309968 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.314329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.310019 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:01.484328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.484291 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edd32a-64b8bf5f55-hk28p"] Nov 01 22:14:01.510740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.510708 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.512618 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.512583 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:01.513229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.513128 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:14:01.513229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:01.513172 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:14:02.271447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.256805 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.321053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.318189 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.321053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.318263 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:02.321053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.319823 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.388496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.365321 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:02.429049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.427705 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.429049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.427758 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:02.565093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.558170 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:14:02.565093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.558237 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:14:02.796548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.784116 4395 generic.go:356] "Generic (PLEG): container finished" podID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerID="df143d6d6101e44c98f46c8e54f31fe502de1ed66ecbf2c623a8521f3b87f765" exitCode=0 Nov 01 22:14:02.796548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:02.791354 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" event={"ID":"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45","Type":"ContainerDied","Data":"df143d6d6101e44c98f46c8e54f31fe502de1ed66ecbf2c623a8521f3b87f765"} Nov 01 22:14:03.132477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.132383 4395 generic.go:356] "Generic (PLEG): container finished" podID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerID="81a0ca7b2b7806acb95f5b36029f5995dc65eead6784739e64a911180ea27934" exitCode=0 Nov 01 22:14:03.153053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.132742 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" event={"ID":"51a30721-9e35-46c5-88c2-47b275f00a7a","Type":"ContainerDied","Data":"81a0ca7b2b7806acb95f5b36029f5995dc65eead6784739e64a911180ea27934"} Nov 01 22:14:03.269429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.269391 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe" path="/var/lib/kubelet/pods/ca03fd1e-f8d3-4c7a-bc29-e569a356c7fe/volumes" Nov 01 22:14:03.352887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.352844 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:14:03.719542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.471599 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:14:03.719542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.618362 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:03.719542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.706245 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:03.719542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:03.706095 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:14:03.722837 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:14:03.722805 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db399d_6ade_4a07_92af_3cfafb61d690.slice/crio-9c1eff50d1d3f2bd1c7e94ec6ccc4d2b1693bb61c57a1dcafa490de47afebd3e WatchSource:0}: Error finding container 9c1eff50d1d3f2bd1c7e94ec6ccc4d2b1693bb61c57a1dcafa490de47afebd3e: Status 404 returned error can't find the container with id 9c1eff50d1d3f2bd1c7e94ec6ccc4d2b1693bb61c57a1dcafa490de47afebd3e Nov 01 22:14:04.050064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.044278 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.050064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.044357 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:04.094501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.093303 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:14:04.094501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.093464 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:14:04.094501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.093546 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:14:04.351366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.351285 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.351604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.351579 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:04.370997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.364831 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:14:04.433591 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.433558 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.433913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.433865 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:04.568989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.568955 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:14:04.569285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.569240 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:14:04.674600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:04.674507 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" event={"ID":"68db399d-6ade-4a07-92af-3cfafb61d690","Type":"ContainerStarted","Data":"9c1eff50d1d3f2bd1c7e94ec6ccc4d2b1693bb61c57a1dcafa490de47afebd3e"} Nov 01 22:14:05.042964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.038564 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:14:05.196119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.196089 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:14:05.219088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.218957 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.226255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.226224 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.226490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.226470 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.226653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.226635 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.226892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.226872 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.227176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.227155 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.232120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.232093 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.232319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.232304 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.232432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.232418 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\") pod \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\" (UID: \"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45\") " Nov 01 22:14:05.242577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.242532 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.246142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.245912 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "kube-api-access-qgwfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:05.246654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.246386 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:05.249845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.249657 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:05.250158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.249975 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.250538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.250292 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.253086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.252845 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.254692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.253226 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.305701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.305450 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:14:05.305701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.305507 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:14:05.345324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.345290 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.346383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.346362 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.346606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.346590 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.346831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.346816 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.346975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.346959 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.347282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.347265 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.347510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.347495 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.347737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.347718 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.348147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.348130 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\") pod \"51a30721-9e35-46c5-88c2-47b275f00a7a\" (UID: \"51a30721-9e35-46c5-88c2-47b275f00a7a\") " Nov 01 22:14:05.349598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.349579 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.349737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.349722 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.349847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.349833 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwfn\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-kube-api-access-qgwfn\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.349966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.349953 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.350087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.350074 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.350209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.350197 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.350314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.350301 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.350423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.350409 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.353851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.353829 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:05.354407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.354379 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.355243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.355218 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.356288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.356101 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.360353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.360168 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.420232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.420195 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:05.420921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.420895 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token" pod "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" (UID: "f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:05.424408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.424379 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:14:05.424511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.424446 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:14:05.431013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.430972 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "kube-api-access-5fvmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:05.433658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.433633 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:05.449777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.441678 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token" pod "51a30721-9e35-46c5-88c2-47b275f00a7a" (UID: "51a30721-9e35-46c5-88c2-47b275f00a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:05.455791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.455763 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.455947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.455931 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvmf\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-kube-api-access-5fvmf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456039 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456141 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51a30721-9e35-46c5-88c2-47b275f00a7a-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456269 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456360 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456438 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456515 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456603 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.456704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.456692 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51a30721-9e35-46c5-88c2-47b275f00a7a-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.800426 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" event={"ID":"68db399d-6ade-4a07-92af-3cfafb61d690","Type":"ContainerStarted","Data":"ed532ffc91cab403612fb2870e4298afabc3be2d2a218fff220a7860e23160ab"} Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.803420 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" event={"ID":"51a30721-9e35-46c5-88c2-47b275f00a7a","Type":"ContainerDied","Data":"a085653746b8b390db92cea6e885a8bc6a76446767a71d0f132eac4956523eb2"} Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.803491 4395 scope.go:122] "RemoveContainer" containerID="81a0ca7b2b7806acb95f5b36029f5995dc65eead6784739e64a911180ea27934" Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.803685 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb" Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.808223 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" Nov 01 22:14:05.830592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.808225 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf" event={"ID":"f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45","Type":"ContainerDied","Data":"28eb0eae96fc8d2a390138a713b4018659b3fc4e40327244e068f136f072c775"} Nov 01 22:14:05.864558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.864524 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:14:05.865487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.865423 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:14:05.873289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.873268 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:14:05.873520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.873506 4395 scope.go:122] "RemoveContainer" containerID="df143d6d6101e44c98f46c8e54f31fe502de1ed66ecbf2c623a8521f3b87f765" Nov 01 22:14:05.883129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.882358 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podStartSLOduration=204.882339885 podStartE2EDuration="3m24.882339885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:05.865476705 +0000 UTC m=+1792.839379241" watchObservedRunningTime="2026-11-01 22:14:05.882339885 +0000 UTC m=+1792.856242425" Nov 01 22:14:05.965498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.965453 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:14:05.980821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:05.980735 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-szbgf"] Nov 01 22:14:06.007017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.006985 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:14:06.011109 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:14:06.011018 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eed9ad_2ef3_48a4_8dc4_e6e51dca6a45.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eed9ad_2ef3_48a4_8dc4_e6e51dca6a45.slice/crio-28eb0eae96fc8d2a390138a713b4018659b3fc4e40327244e068f136f072c775\": RecentStats: unable to find data in memory cache]" Nov 01 22:14:06.015409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.015386 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-8vrhb"] Nov 01 22:14:06.275077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.274611 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:06.275935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.275909 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:06.276108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.275965 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:06.313467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.313420 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:14:06.314499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.314469 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:14:06.315493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.315474 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:14:06.428248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.428222 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:14:06.429376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.429337 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:14:06.430311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:06.430291 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:14:07.168009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.167969 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" path="/var/lib/kubelet/pods/51a30721-9e35-46c5-88c2-47b275f00a7a/volumes" Nov 01 22:14:07.168650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.168623 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" path="/var/lib/kubelet/pods/f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45/volumes" Nov 01 22:14:07.295463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.295421 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:07.295742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.295712 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:07.407144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.407106 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:14:07.408475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408445 4395 cpu_manager.go:414] "containerMap: removing container" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" Nov 01 22:14:07.408475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408472 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" Nov 01 22:14:07.408657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408533 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" Nov 01 22:14:07.408657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408542 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" Nov 01 22:14:07.408752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408739 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a30721-9e35-46c5-88c2-47b275f00a7a" containerName="istio-proxy" Nov 01 22:14:07.408803 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.408764 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eed9ad-2ef3-48a4-8dc4-e6e51dca6a45" containerName="istio-proxy" Nov 01 22:14:07.414598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.414571 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.414780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.414745 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:14:07.425290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.421692 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7e2df\"" Nov 01 22:14:07.425290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.424977 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e7e2df-dockercfg-fbtgr\"" Nov 01 22:14:07.436160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.436135 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:14:07.443685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.443651 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthvr\" (UniqueName: \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.443799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.443759 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.445302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.445281 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:14:07.445475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.445388 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:07.561055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.558474 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthvr\" (UniqueName: \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.561055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.558591 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.561055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.559839 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.672077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.661425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:07.672077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.661873 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmrx\" (UniqueName: \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:07.855052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.852491 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthvr\" (UniqueName: \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\") pod \"payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.876061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.874470 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:07.944050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.878008 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmrx\" (UniqueName: \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:07.964931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.951427 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:07.964931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:07.962951 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:08.046554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.046525 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmrx\" (UniqueName: \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\") pod \"payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:08.064391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.064180 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:08.306051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.305499 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:08.306051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.305563 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:08.361469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.361200 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:14:08.436090 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:14:08.436053 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed9619c_ae76_4c69_b97f_43a01a181cee.slice/crio-fcb276afe11016ed110bc854f1f8f4603a03463283f1c273acec1fa341db5907 WatchSource:0}: Error finding container fcb276afe11016ed110bc854f1f8f4603a03463283f1c273acec1fa341db5907: Status 404 returned error can't find the container with id fcb276afe11016ed110bc854f1f8f4603a03463283f1c273acec1fa341db5907 Nov 01 22:14:08.436224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:08.436114 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:14:09.053235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.053184 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" event={"ID":"97e71ccc-afcb-47b8-b973-da12247405fb","Type":"ContainerStarted","Data":"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37"} Nov 01 22:14:09.053441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.053255 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" event={"ID":"97e71ccc-afcb-47b8-b973-da12247405fb","Type":"ContainerStarted","Data":"b029abcec1e017c742a2b82f4836972be8731e09631678a76503a6fb6afc4fc3"} Nov 01 22:14:09.053441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.053320 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:09.054974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.054944 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" event={"ID":"0ed9619c-ae76-4c69-b97f-43a01a181cee","Type":"ContainerStarted","Data":"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4"} Nov 01 22:14:09.055112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.054987 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" event={"ID":"0ed9619c-ae76-4c69-b97f-43a01a181cee","Type":"ContainerStarted","Data":"fcb276afe11016ed110bc854f1f8f4603a03463283f1c273acec1fa341db5907"} Nov 01 22:14:09.055112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.055063 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:09.082053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.074647 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" podStartSLOduration=2.074629377 podStartE2EDuration="2.074629377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:09.071057822 +0000 UTC m=+1796.044960355" watchObservedRunningTime="2026-11-01 22:14:09.074629377 +0000 UTC m=+1796.048531917" Nov 01 22:14:09.277054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.276850 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:09.277054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:09.276916 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:10.300055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:10.296433 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:10.300055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:10.296498 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:11.281895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:11.281451 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:11.281895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:11.281514 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:12.275114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:12.275065 4395 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:14:12.275693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:12.275137 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:14:13.280012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:13.279979 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:13.280665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:13.280509 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:13.281573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:13.281548 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:14:13.308999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:13.308951 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" podStartSLOduration=6.30893562 podStartE2EDuration="6.30893562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:09.108272292 +0000 UTC m=+1796.082174831" watchObservedRunningTime="2026-11-01 22:14:13.30893562 +0000 UTC m=+1800.282838165" Nov 01 22:14:20.067693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:20.067663 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:14:20.068113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:20.067749 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:14:22.305993 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:14:22.305948 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:14:26.250019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.249978 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:14:26.258727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.258702 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.266648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.266627 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7e2df-serving-cert\"" Nov 01 22:14:26.270482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.270194 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:14:26.400057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.392508 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.400057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.392636 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.400057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.392757 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdxd\" (UniqueName: \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.411236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.411195 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.515353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.513309 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdxd\" (UniqueName: \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.515353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.513579 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.515353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.513728 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.515353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.513811 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.515353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.514746 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.523073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.523013 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.524797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.524771 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.532629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.532574 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdxd\" (UniqueName: \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\") pod \"maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.581792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.581769 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:26.977908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:26.977802 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:14:27.006830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:27.005720 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:14:27.455941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:27.455906 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" event={"ID":"ad8bc2bc-648f-48d3-a826-498915ea3add","Type":"ContainerStarted","Data":"4ddd15946f3d3de2eeb4a56d6a4c1261d1c108a9c0349de5752cbbefb66a501b"} Nov 01 22:14:28.479616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:28.479562 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" event={"ID":"ad8bc2bc-648f-48d3-a826-498915ea3add","Type":"ContainerStarted","Data":"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe"} Nov 01 22:14:28.480172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:28.479875 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:28.504985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:28.504934 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" podStartSLOduration=1.9409359579999999 podStartE2EDuration="2.504921083s" totalImagesPullingTime="563.985125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:28.500683921 +0000 UTC m=+1815.474586461" watchObservedRunningTime="2026-11-01 22:14:28.504921083 +0000 UTC m=+1815.478823622" Nov 01 22:14:29.225875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:29.210368 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:14:29.225875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:29.210603 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" containerID="cri-o://03b2ebfdc40ff5e5d22ab94e2e9e8f88c11390db2ee88ec008ca7cd2ff7835a0" gracePeriod=30 Nov 01 22:14:34.585047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:34.582199 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:14:34.805317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:34.804858 4395 generic.go:356] "Generic (PLEG): container finished" podID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerID="03b2ebfdc40ff5e5d22ab94e2e9e8f88c11390db2ee88ec008ca7cd2ff7835a0" exitCode=0 Nov 01 22:14:34.805317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:34.805218 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" event={"ID":"cfb96a1f-ea6e-446a-b60a-cccccc414a3b","Type":"ContainerDied","Data":"03b2ebfdc40ff5e5d22ab94e2e9e8f88c11390db2ee88ec008ca7cd2ff7835a0"} Nov 01 22:14:34.957963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:34.957933 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:14:35.105953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.105917 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.105990 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106209 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106223 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.106333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106292 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.106333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106372 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106414 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106498 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.106646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106626 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106627 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.106709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106678 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.106829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.106729 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\") pod \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\" (UID: \"cfb96a1f-ea6e-446a-b60a-cccccc414a3b\") " Nov 01 22:14:35.107308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107278 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:14:35.107818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107781 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.107818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107815 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.107990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107831 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.107990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107845 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.107990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.107857 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.108876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.108848 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "kube-api-access-7j9lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:35.109072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.109009 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:14:35.109549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.109524 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:14:35.109640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.109573 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token" pod "cfb96a1f-ea6e-446a-b60a-cccccc414a3b" (UID: "cfb96a1f-ea6e-446a-b60a-cccccc414a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:14:35.210679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.210644 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9lw\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-kube-api-access-7j9lw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.210833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.210683 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.210833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.210700 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.210833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.210715 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb96a1f-ea6e-446a-b60a-cccccc414a3b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:14:35.665634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.665603 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:14:35.666472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.666455 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" Nov 01 22:14:35.666537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.666474 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" Nov 01 22:14:35.666652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.666641 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" containerName="istio-proxy" Nov 01 22:14:35.671506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.671487 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.680283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.680257 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:14:35.681536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.681514 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-d6e25f\"" Nov 01 22:14:35.682565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.682519 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-d6e25f-dockercfg-f5whc\"" Nov 01 22:14:35.688457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.688432 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:14:35.693842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.693821 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.717438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.717410 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:14:35.729071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.722271 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44w9f\" (UniqueName: \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.729071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.722425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.729071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.722487 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncslh\" (UniqueName: \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.729071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.722634 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.912952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.913153 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44w9f\" (UniqueName: \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.913300 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.913379 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncslh\" (UniqueName: \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.914682 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:35.925199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.915866 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:35.980048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.979995 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" event={"ID":"cfb96a1f-ea6e-446a-b60a-cccccc414a3b","Type":"ContainerDied","Data":"982e9fee8cceac454c21855baa6c2f16d9ccc9e9c962950798dcdf14df56a0de"} Nov 01 22:14:35.980335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.980319 4395 scope.go:122] "RemoveContainer" containerID="03b2ebfdc40ff5e5d22ab94e2e9e8f88c11390db2ee88ec008ca7cd2ff7835a0" Nov 01 22:14:35.980615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:35.980597 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc" Nov 01 22:14:36.081858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.081824 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncslh\" (UniqueName: \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\") pod \"payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:36.091262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.091192 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:14:36.096075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.096014 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-8gngc"] Nov 01 22:14:36.101839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.101806 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44w9f\" (UniqueName: \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\") pod \"payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:36.289449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.289405 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:36.321157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.321129 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:36.597922 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:14:36.597887 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d8dbb8_1d5b_4640_93f9_c1f2763ff2d4.slice/crio-f7feb451a9fea816a733cd9c4e20fee41b242477996f5b1af07b264e532068a6 WatchSource:0}: Error finding container f7feb451a9fea816a733cd9c4e20fee41b242477996f5b1af07b264e532068a6: Status 404 returned error can't find the container with id f7feb451a9fea816a733cd9c4e20fee41b242477996f5b1af07b264e532068a6 Nov 01 22:14:36.598883 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.598859 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:14:36.632852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.632819 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:14:36.637570 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:14:36.637546 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9538ac7_9b20_46f9_a6fc_d0fef2bbacc5.slice/crio-bd8119d612de46fa56e4b32923b85578066a685a4d78852eb3145e7a34c03607 WatchSource:0}: Error finding container bd8119d612de46fa56e4b32923b85578066a685a4d78852eb3145e7a34c03607: Status 404 returned error can't find the container with id bd8119d612de46fa56e4b32923b85578066a685a4d78852eb3145e7a34c03607 Nov 01 22:14:36.992943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.992834 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" event={"ID":"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5","Type":"ContainerStarted","Data":"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761"} Nov 01 22:14:36.992943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.992907 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" event={"ID":"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5","Type":"ContainerStarted","Data":"bd8119d612de46fa56e4b32923b85578066a685a4d78852eb3145e7a34c03607"} Nov 01 22:14:36.993490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.992957 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:36.994983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.994953 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" event={"ID":"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4","Type":"ContainerStarted","Data":"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f"} Nov 01 22:14:36.995124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.995002 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" event={"ID":"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4","Type":"ContainerStarted","Data":"f7feb451a9fea816a733cd9c4e20fee41b242477996f5b1af07b264e532068a6"} Nov 01 22:14:36.995124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:36.995105 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:37.015953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:37.015899 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" podStartSLOduration=2.015884396 podStartE2EDuration="2.015884396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:37.011917286 +0000 UTC m=+1823.985819826" watchObservedRunningTime="2026-11-01 22:14:37.015884396 +0000 UTC m=+1823.989786935" Nov 01 22:14:37.034455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:37.034391 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" podStartSLOduration=2.034372789 podStartE2EDuration="2.034372789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:37.031600978 +0000 UTC m=+1824.005503518" watchObservedRunningTime="2026-11-01 22:14:37.034372789 +0000 UTC m=+1824.008275330" Nov 01 22:14:37.273054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:37.271627 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb96a1f-ea6e-446a-b60a-cccccc414a3b" path="/var/lib/kubelet/pods/cfb96a1f-ea6e-446a-b60a-cccccc414a3b/volumes" Nov 01 22:14:48.027053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:48.024214 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:14:48.027053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:14:48.025793 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:14:52.306366 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:14:52.306330 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:15:00.177487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.177387 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:15:00.178254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.177673 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" containerID="cri-o://2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3" gracePeriod=30 Nov 01 22:15:00.190388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.190352 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:15:00.213745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.213710 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:00.230827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.229325 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.246097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.246000 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:00.261616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.253535 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.261616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.255981 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:15:00.266749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.266713 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:00.275104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.275073 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:00.279494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.279467 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.280105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.280076 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6v79\" (UniqueName: \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.301047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.300661 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:00.311635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.311608 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.315242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.311484 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:00.323308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.323282 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:00.323644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.323627 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.333832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.333623 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:00.343063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.342741 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:00.359357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.359328 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.371326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.371294 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:00.403061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403008 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjh7\" (UniqueName: \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.403383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403352 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.403558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403441 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhs5\" (UniqueName: \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.403697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403677 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.403837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403812 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gz2h\" (UniqueName: \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.403936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403912 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.404000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.403976 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6v79\" (UniqueName: \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.404147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.404126 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.404217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.404154 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.416860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.416824 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6v79\" (UniqueName: \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.505809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.505775 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.506017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.505848 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gz2h\" (UniqueName: \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.506017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.505866 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.506017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.505980 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jc7p\" (UniqueName: \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.506285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.506285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506221 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.506285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506255 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjh7\" (UniqueName: \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.506463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506366 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhs5\" (UniqueName: \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.506649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506618 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.506737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506652 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.506846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.506831 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.518481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.518451 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhs5\" (UniqueName: \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.518606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.518522 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gz2h\" (UniqueName: \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.518606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.518523 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjh7\" (UniqueName: \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.567042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.567005 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:00.608002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.607965 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jc7p\" (UniqueName: \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.608216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.608145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.608832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.608809 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.613632 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.613575 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:00.626299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.621580 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jc7p\" (UniqueName: \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.772426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.761715 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:00.772426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.764415 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:00.776410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.776369 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:00.792462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.792422 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" containerID="cri-o://72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901" gracePeriod=30 Nov 01 22:15:00.870157 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:00.870102 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e0c015_1ebc_4d45_b409_0aadc0b1b7b4.slice/crio-2d08ecbee5b2b5a34983e72c19c609606f76f993d050a85d3cb799534e7a39a7 WatchSource:0}: Error finding container 2d08ecbee5b2b5a34983e72c19c609606f76f993d050a85d3cb799534e7a39a7: Status 404 returned error can't find the container with id 2d08ecbee5b2b5a34983e72c19c609606f76f993d050a85d3cb799534e7a39a7 Nov 01 22:15:00.871514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:00.871480 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:01.055563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.055534 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:01.077671 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:01.077636 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bee970_09ad_4a69_b3e9_79c68b3313f8.slice/crio-39b46438b974e1f260eca50efd2966722c2a691d6a1b3bf06ffae02d2af53b49 WatchSource:0}: Error finding container 39b46438b974e1f260eca50efd2966722c2a691d6a1b3bf06ffae02d2af53b49: Status 404 returned error can't find the container with id 39b46438b974e1f260eca50efd2966722c2a691d6a1b3bf06ffae02d2af53b49 Nov 01 22:15:01.135683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.135655 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:01.142954 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:01.142926 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26ee628_29b5_418f_9009_a9b75b08fc87.slice/crio-9013bbbd06d0ab680364ce7485f7b1ab41dc53657ab97e816cfa21c5a2a75bbd WatchSource:0}: Error finding container 9013bbbd06d0ab680364ce7485f7b1ab41dc53657ab97e816cfa21c5a2a75bbd: Status 404 returned error can't find the container with id 9013bbbd06d0ab680364ce7485f7b1ab41dc53657ab97e816cfa21c5a2a75bbd Nov 01 22:15:01.189585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.189556 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:01.190113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.190092 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:01.801637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.801601 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerStarted","Data":"1db68d971f32da8cc19789ab34d5dcbf01bf4a38d71227f915f8cae71d93f866"} Nov 01 22:15:01.803447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.803420 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerStarted","Data":"e6914bebddbec7d213f3edec6d1e750b3c15d11022f7c5f3149b30440c2292c9"} Nov 01 22:15:01.803585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.803474 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerStarted","Data":"9013bbbd06d0ab680364ce7485f7b1ab41dc53657ab97e816cfa21c5a2a75bbd"} Nov 01 22:15:01.805228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.805202 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerStarted","Data":"38172e21a37955eb893a901307dacfbef621a93329e83798148d08472a5bb574"} Nov 01 22:15:01.805325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.805254 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerStarted","Data":"39b46438b974e1f260eca50efd2966722c2a691d6a1b3bf06ffae02d2af53b49"} Nov 01 22:15:01.807263 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.807240 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerStarted","Data":"a7f111c378f8905991037bf12f6177318b918e2ad9e729a29a28c7cd203d86ff"} Nov 01 22:15:01.807371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.807281 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerStarted","Data":"2d08ecbee5b2b5a34983e72c19c609606f76f993d050a85d3cb799534e7a39a7"} Nov 01 22:15:01.809117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.809096 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerStarted","Data":"a16d534b9b78e573c57a7f92d07255939dfa88435916d9f10d21e24ad1a8fc0b"} Nov 01 22:15:01.809201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.809137 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerStarted","Data":"027b67051402d8934dda4478a9bb96b861fe6c8640fa1ad7d530b7800c08986a"} Nov 01 22:15:01.824156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.824081 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" podStartSLOduration=1.824068711 podStartE2EDuration="1.824068711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:01.820016859 +0000 UTC m=+1848.793919397" watchObservedRunningTime="2026-11-01 22:15:01.824068711 +0000 UTC m=+1848.797971252" Nov 01 22:15:01.841287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.841240 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" podStartSLOduration=1.8412259739999999 podStartE2EDuration="1.841225974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:01.836876124 +0000 UTC m=+1848.810778668" watchObservedRunningTime="2026-11-01 22:15:01.841225974 +0000 UTC m=+1848.815128703" Nov 01 22:15:01.855861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.855810 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" podStartSLOduration=1.8557970350000002 podStartE2EDuration="1.855797035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:01.854351564 +0000 UTC m=+1848.828254125" watchObservedRunningTime="2026-11-01 22:15:01.855797035 +0000 UTC m=+1848.829699633" Nov 01 22:15:01.882334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:01.882283 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" podStartSLOduration=1.882267629 podStartE2EDuration="1.882267629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:01.872466918 +0000 UTC m=+1848.846369458" watchObservedRunningTime="2026-11-01 22:15:01.882267629 +0000 UTC m=+1848.856170168" Nov 01 22:15:02.819298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.819260 4395 generic.go:356] "Generic (PLEG): container finished" podID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerID="a7f111c378f8905991037bf12f6177318b918e2ad9e729a29a28c7cd203d86ff" exitCode=60 Nov 01 22:15:02.819739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.819314 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerDied","Data":"a7f111c378f8905991037bf12f6177318b918e2ad9e729a29a28c7cd203d86ff"} Nov 01 22:15:02.819800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.819736 4395 scope.go:122] "RemoveContainer" containerID="a7f111c378f8905991037bf12f6177318b918e2ad9e729a29a28c7cd203d86ff" Nov 01 22:15:02.821343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.821320 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d4211a2-1465-497f-9165-70f935c78220" containerID="a16d534b9b78e573c57a7f92d07255939dfa88435916d9f10d21e24ad1a8fc0b" exitCode=60 Nov 01 22:15:02.821471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.821399 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerDied","Data":"a16d534b9b78e573c57a7f92d07255939dfa88435916d9f10d21e24ad1a8fc0b"} Nov 01 22:15:02.821693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.821676 4395 scope.go:122] "RemoveContainer" containerID="a16d534b9b78e573c57a7f92d07255939dfa88435916d9f10d21e24ad1a8fc0b" Nov 01 22:15:02.824627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.823817 4395 generic.go:356] "Generic (PLEG): container finished" podID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerID="3627e6d34a7c0037319b83de72e3e13c33b80d8e7adc57d74405c33681967d53" exitCode=60 Nov 01 22:15:02.824627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.823923 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerDied","Data":"3627e6d34a7c0037319b83de72e3e13c33b80d8e7adc57d74405c33681967d53"} Nov 01 22:15:02.824627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.824246 4395 scope.go:122] "RemoveContainer" containerID="3627e6d34a7c0037319b83de72e3e13c33b80d8e7adc57d74405c33681967d53" Nov 01 22:15:02.825954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.825817 4395 generic.go:356] "Generic (PLEG): container finished" podID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerID="e6914bebddbec7d213f3edec6d1e750b3c15d11022f7c5f3149b30440c2292c9" exitCode=60 Nov 01 22:15:02.825954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.825895 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerDied","Data":"e6914bebddbec7d213f3edec6d1e750b3c15d11022f7c5f3149b30440c2292c9"} Nov 01 22:15:02.826164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.826145 4395 scope.go:122] "RemoveContainer" containerID="e6914bebddbec7d213f3edec6d1e750b3c15d11022f7c5f3149b30440c2292c9" Nov 01 22:15:02.827565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.827544 4395 generic.go:356] "Generic (PLEG): container finished" podID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerID="38172e21a37955eb893a901307dacfbef621a93329e83798148d08472a5bb574" exitCode=60 Nov 01 22:15:02.827648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.827610 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerDied","Data":"38172e21a37955eb893a901307dacfbef621a93329e83798148d08472a5bb574"} Nov 01 22:15:02.827796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:02.827783 4395 scope.go:122] "RemoveContainer" containerID="38172e21a37955eb893a901307dacfbef621a93329e83798148d08472a5bb574" Nov 01 22:15:03.727212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.727132 4395 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:15:03.727212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.727195 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:15:03.861567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.861473 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerStarted","Data":"db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d"} Nov 01 22:15:03.863941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.863913 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerStarted","Data":"0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c"} Nov 01 22:15:03.866715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.866684 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerStarted","Data":"a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69"} Nov 01 22:15:03.874736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.874699 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerStarted","Data":"d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f"} Nov 01 22:15:03.878466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.878440 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerStarted","Data":"a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856"} Nov 01 22:15:03.948574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:03.948518 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" podStartSLOduration=3.9485026899999998 podStartE2EDuration="3.94850269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:03.943781304 +0000 UTC m=+1850.917683847" watchObservedRunningTime="2026-11-01 22:15:03.94850269 +0000 UTC m=+1850.922405229" Nov 01 22:15:04.889230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.889196 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d4211a2-1465-497f-9165-70f935c78220" containerID="0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c" exitCode=60 Nov 01 22:15:04.889684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.889277 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerDied","Data":"0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c"} Nov 01 22:15:04.889684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.889361 4395 scope.go:122] "RemoveContainer" containerID="a16d534b9b78e573c57a7f92d07255939dfa88435916d9f10d21e24ad1a8fc0b" Nov 01 22:15:04.889684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.889562 4395 scope.go:122] "RemoveContainer" containerID="0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c" Nov 01 22:15:04.890147 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:04.890105 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn_odh-ai-gateway-infra(6d4211a2-1465-497f-9165-70f935c78220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" podUID="6d4211a2-1465-497f-9165-70f935c78220" Nov 01 22:15:04.892016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.891995 4395 generic.go:356] "Generic (PLEG): container finished" podID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerID="a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69" exitCode=60 Nov 01 22:15:04.892160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.892131 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerDied","Data":"a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69"} Nov 01 22:15:04.892390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.892375 4395 scope.go:122] "RemoveContainer" containerID="a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69" Nov 01 22:15:04.892776 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:04.892753 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr_odh-ai-gateway-infra(908fe8ef-cb40-4b03-9957-56ef46f9d6aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" Nov 01 22:15:04.894175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.894151 4395 generic.go:356] "Generic (PLEG): container finished" podID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerID="d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f" exitCode=60 Nov 01 22:15:04.894262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.894188 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerDied","Data":"d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f"} Nov 01 22:15:04.894464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.894445 4395 scope.go:122] "RemoveContainer" containerID="d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f" Nov 01 22:15:04.894738 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:04.894719 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4_odh-ai-gateway-infra(d26ee628-29b5-418f-9009-a9b75b08fc87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" Nov 01 22:15:04.896262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.896241 4395 generic.go:356] "Generic (PLEG): container finished" podID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerID="a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856" exitCode=60 Nov 01 22:15:04.896333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.896262 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerDied","Data":"a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856"} Nov 01 22:15:04.896538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.896523 4395 scope.go:122] "RemoveContainer" containerID="a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856" Nov 01 22:15:04.896779 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:04.896758 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x_odh-ai-gateway-infra(e8bee970-09ad-4a69-b3e9-79c68b3313f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" Nov 01 22:15:04.898733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.898711 4395 generic.go:356] "Generic (PLEG): container finished" podID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerID="db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d" exitCode=60 Nov 01 22:15:04.898889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.898749 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerDied","Data":"db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d"} Nov 01 22:15:04.899089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.899069 4395 scope.go:122] "RemoveContainer" containerID="db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d" Nov 01 22:15:04.899353 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:04.899330 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786_odh-ai-gateway-infra(a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" Nov 01 22:15:04.904183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.904161 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2"] Nov 01 22:15:04.905839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.905820 4395 scope.go:122] "RemoveContainer" containerID="3627e6d34a7c0037319b83de72e3e13c33b80d8e7adc57d74405c33681967d53" Nov 01 22:15:04.910551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.910534 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:04.915720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.915701 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:04.922540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.922516 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2"] Nov 01 22:15:04.932753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.931208 4395 scope.go:122] "RemoveContainer" containerID="e6914bebddbec7d213f3edec6d1e750b3c15d11022f7c5f3149b30440c2292c9" Nov 01 22:15:04.949067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.949006 4395 scope.go:122] "RemoveContainer" containerID="38172e21a37955eb893a901307dacfbef621a93329e83798148d08472a5bb574" Nov 01 22:15:04.966856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:04.966833 4395 scope.go:122] "RemoveContainer" containerID="a7f111c378f8905991037bf12f6177318b918e2ad9e729a29a28c7cd203d86ff" Nov 01 22:15:05.023006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.022968 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.023228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.023203 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgqg\" (UniqueName: \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.023551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.023530 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.124986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.124955 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.125207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.125188 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.125290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.125249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgqg\" (UniqueName: \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.125643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.125615 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.127660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.127632 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.138000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.137981 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgqg\" (UniqueName: \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\") pod \"maas-api-revoke-sub-a791869f-z5cx2\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.236518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.236491 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:05.534830 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:05.534800 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5be401_9cd6_4f3c_a6c7_0dcfa2cfc40d.slice/crio-af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36 WatchSource:0}: Error finding container af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36: Status 404 returned error can't find the container with id af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36 Nov 01 22:15:05.535676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.535646 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2"] Nov 01 22:15:05.618143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.618117 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:15:05.747258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747226 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747322 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747405 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747437 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747471 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747552 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747636 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747756 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.747847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.747833 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\") pod \"7642982a-fd39-4afc-b878-798249a7a0cd\" (UID: \"7642982a-fd39-4afc-b878-798249a7a0cd\") " Nov 01 22:15:05.748121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.748081 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:05.748395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.748369 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:05.748515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.748413 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:05.748667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.748649 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:05.749420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.749394 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.749499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.749426 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.749499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.749443 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.749499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.749458 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7642982a-fd39-4afc-b878-798249a7a0cd-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.749738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.749717 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:05.750363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.750343 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "kube-api-access-jndr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:05.750453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.750378 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:05.750453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.750439 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:05.750545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.750524 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo" pod "7642982a-fd39-4afc-b878-798249a7a0cd" (UID: "7642982a-fd39-4afc-b878-798249a7a0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:05.851473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.851441 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7642982a-fd39-4afc-b878-798249a7a0cd-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.851473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.851470 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.851656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.851480 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndr8\" (UniqueName: \"kubernetes.io/projected/7642982a-fd39-4afc-b878-798249a7a0cd-kube-api-access-jndr8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.851656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.851491 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.851656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.851501 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7642982a-fd39-4afc-b878-798249a7a0cd-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:05.909235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.909201 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" event={"ID":"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d","Type":"ContainerStarted","Data":"af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36"} Nov 01 22:15:05.911637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.911611 4395 scope.go:122] "RemoveContainer" containerID="a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69" Nov 01 22:15:05.912129 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.912096 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr_odh-ai-gateway-infra(908fe8ef-cb40-4b03-9957-56ef46f9d6aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" Nov 01 22:15:05.913000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.912971 4395 generic.go:356] "Generic (PLEG): container finished" podID="7642982a-fd39-4afc-b878-798249a7a0cd" containerID="2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3" exitCode=0 Nov 01 22:15:05.913141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.913040 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" Nov 01 22:15:05.913141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.913060 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerDied","Data":"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3"} Nov 01 22:15:05.913141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.913118 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4" event={"ID":"7642982a-fd39-4afc-b878-798249a7a0cd","Type":"ContainerDied","Data":"92f38979cbaca9f9627dd5bea6fbefdc996acb00baabefac2e312ebc07e907d9"} Nov 01 22:15:05.913305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.913149 4395 scope.go:122] "RemoveContainer" containerID="2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3" Nov 01 22:15:05.915667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.915647 4395 scope.go:122] "RemoveContainer" containerID="d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f" Nov 01 22:15:05.916141 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.916006 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4_odh-ai-gateway-infra(d26ee628-29b5-418f-9009-a9b75b08fc87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" Nov 01 22:15:05.917626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.917611 4395 scope.go:122] "RemoveContainer" containerID="a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856" Nov 01 22:15:05.917943 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.917924 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x_odh-ai-gateway-infra(e8bee970-09ad-4a69-b3e9-79c68b3313f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" Nov 01 22:15:05.919807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.919785 4395 scope.go:122] "RemoveContainer" containerID="db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d" Nov 01 22:15:05.920140 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.920108 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786_odh-ai-gateway-infra(a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" Nov 01 22:15:05.922042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.922005 4395 scope.go:122] "RemoveContainer" containerID="0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c" Nov 01 22:15:05.922361 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.922338 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn_odh-ai-gateway-infra(6d4211a2-1465-497f-9165-70f935c78220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" podUID="6d4211a2-1465-497f-9165-70f935c78220" Nov 01 22:15:05.926645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.926625 4395 scope.go:122] "RemoveContainer" containerID="10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262" Nov 01 22:15:05.940541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.940516 4395 scope.go:122] "RemoveContainer" containerID="2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3" Nov 01 22:15:05.940820 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.940795 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3\": container with ID starting with 2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3 not found: ID does not exist" containerID="2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3" Nov 01 22:15:05.940899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.940827 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3"} err="failed to get container status \"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3\": rpc error: code = NotFound desc = could not find container \"2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3\": container with ID starting with 2389389cf46e8dfae9fb2ccca26b9b74df9c85fc7ff4765088e86dabf765f4a3 not found: ID does not exist" Nov 01 22:15:05.940899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.940842 4395 scope.go:122] "RemoveContainer" containerID="10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262" Nov 01 22:15:05.941096 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:05.941079 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262\": container with ID starting with 10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262 not found: ID does not exist" containerID="10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262" Nov 01 22:15:05.941178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.941101 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262"} err="failed to get container status \"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262\": rpc error: code = NotFound desc = could not find container \"10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262\": container with ID starting with 10f165e3f7c0ac0fcd9b609023394619eb1c12e38b467241d03b7847a3376262 not found: ID does not exist" Nov 01 22:15:05.953306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.953274 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:15:05.965382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:05.964422 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-47sm4"] Nov 01 22:15:06.097442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.097418 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.267471 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.267655 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269147 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269256 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269309 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269350 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269497 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269562 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269611 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.269737 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\") pod \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\" (UID: \"d76a221b-6d77-4f50-8e74-59d8e0a1f862\") " Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.271491 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.271897 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.272242 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.272624 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.273374 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.273400 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.273415 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.273429 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.278143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.273443 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.355267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.343224 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:06.364055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.360163 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:06.364055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.360261 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:06.377989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.377905 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.377989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.377941 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.377989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.377955 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d76a221b-6d77-4f50-8e74-59d8e0a1f862-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.378547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.378520 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr" pod "d76a221b-6d77-4f50-8e74-59d8e0a1f862" (UID: "d76a221b-6d77-4f50-8e74-59d8e0a1f862"). InnerVolumeSpecName "kube-api-access-pk4fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:06.480068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.480018 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk4fr\" (UniqueName: \"kubernetes.io/projected/d76a221b-6d77-4f50-8e74-59d8e0a1f862-kube-api-access-pk4fr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:06.932663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.932634 4395 generic.go:356] "Generic (PLEG): container finished" podID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerID="72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901" exitCode=0 Nov 01 22:15:06.933088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.932695 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" Nov 01 22:15:06.933088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.932730 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" event={"ID":"d76a221b-6d77-4f50-8e74-59d8e0a1f862","Type":"ContainerDied","Data":"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901"} Nov 01 22:15:06.933088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.932789 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz" event={"ID":"d76a221b-6d77-4f50-8e74-59d8e0a1f862","Type":"ContainerDied","Data":"42a1a8be3edeaf0f107c684c5926051506381a175e2ce8cab04b218df0d09a04"} Nov 01 22:15:06.933088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.932819 4395 scope.go:122] "RemoveContainer" containerID="72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901" Nov 01 22:15:06.934698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.934675 4395 generic.go:356] "Generic (PLEG): container finished" podID="2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" containerID="f0ebd277c69961953d62e6dede4c6c366df22c33e05b498e013e0f0cf6c539c3" exitCode=0 Nov 01 22:15:06.934811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.934737 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" event={"ID":"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d","Type":"ContainerDied","Data":"f0ebd277c69961953d62e6dede4c6c366df22c33e05b498e013e0f0cf6c539c3"} Nov 01 22:15:06.957248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.957084 4395 scope.go:122] "RemoveContainer" containerID="72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901" Nov 01 22:15:06.958656 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:06.958636 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901\": container with ID starting with 72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901 not found: ID does not exist" containerID="72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901" Nov 01 22:15:06.958748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.958664 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901"} err="failed to get container status \"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901\": rpc error: code = NotFound desc = could not find container \"72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901\": container with ID starting with 72909c71fb5e48a9ec8b0fe7ad4c5f119ca970e0288d32161608f20771b5e901 not found: ID does not exist" Nov 01 22:15:06.976571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.976545 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:15:06.980890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:06.980865 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-55jsz"] Nov 01 22:15:07.267282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.265964 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" path="/var/lib/kubelet/pods/7642982a-fd39-4afc-b878-798249a7a0cd/volumes" Nov 01 22:15:07.267282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.267007 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" path="/var/lib/kubelet/pods/d76a221b-6d77-4f50-8e74-59d8e0a1f862/volumes" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.972556 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.973988 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974014 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974113 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974124 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974168 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974180 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974687 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974712 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76a221b-6d77-4f50-8e74-59d8e0a1f862" containerName="istio-proxy" Nov 01 22:15:07.983162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:07.974730 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7642982a-fd39-4afc-b878-798249a7a0cd" containerName="istio-proxy" Nov 01 22:15:08.022107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.022073 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.024673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.024579 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:15:08.096825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.095681 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8483c2-openshift-default-dockercfg-nw2v7\"" Nov 01 22:15:08.119271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119237 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119331 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119479 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119592 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119798 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119888 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119945 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtl6\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.121120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.119983 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221343 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221461 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221527 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221591 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtl6\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221653 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221716 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221743 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.221839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.221785 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.222747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.222721 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.224394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.224372 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.225533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.225506 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.225644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.225616 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.225797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.225767 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.228108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.228082 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.231474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.231452 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.234078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.234006 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.237620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.237596 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtl6\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\") pod \"e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.276212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.276190 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:08.404716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.404680 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:08.425508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.425472 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\") pod \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " Nov 01 22:15:08.425633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.425580 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\") pod \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " Nov 01 22:15:08.425860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.425833 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca" pod "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" (UID: "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:08.425860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.425854 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\" (UniqueName: \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\") pod \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\" (UID: \"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d\") " Nov 01 22:15:08.426667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.426641 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:08.428059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.428015 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca" pod "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" (UID: "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:08.428524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.428500 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg" pod "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" (UID: "2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d"). InnerVolumeSpecName "kube-api-access-lpgqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:08.528762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.528721 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgqg\" (UniqueName: \"kubernetes.io/projected/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-kube-api-access-lpgqg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:08.528933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.528767 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:08.745090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.738503 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:15:08.750170 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:08.748739 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5d8060_ec8b_4828_95f2_6da59624e20b.slice/crio-88235f6cbe1811e7098308e8c7aaf80811240d88cd7c0a7ced16175ceeb1c8fa WatchSource:0}: Error finding container 88235f6cbe1811e7098308e8c7aaf80811240d88cd7c0a7ced16175ceeb1c8fa: Status 404 returned error can't find the container with id 88235f6cbe1811e7098308e8c7aaf80811240d88cd7c0a7ced16175ceeb1c8fa Nov 01 22:15:08.753356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.752770 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:08.753356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.752911 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:08.753356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.752955 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:08.996561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.996530 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" Nov 01 22:15:08.997053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.996523 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2" event={"ID":"2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d","Type":"ContainerDied","Data":"af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36"} Nov 01 22:15:08.997053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.996655 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af258b177f2279a9dbb233c6e20f945a3549909e84c38ad210f5929d52b25d36" Nov 01 22:15:08.998553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.998525 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" event={"ID":"1c5d8060-ec8b-4828-95f2-6da59624e20b","Type":"ContainerStarted","Data":"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15"} Nov 01 22:15:08.998719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:08.998567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" event={"ID":"1c5d8060-ec8b-4828-95f2-6da59624e20b","Type":"ContainerStarted","Data":"88235f6cbe1811e7098308e8c7aaf80811240d88cd7c0a7ced16175ceeb1c8fa"} Nov 01 22:15:09.025274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.025230 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podStartSLOduration=2.025215542 podStartE2EDuration="2.025215542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:09.021474558 +0000 UTC m=+1855.995377097" watchObservedRunningTime="2026-11-01 22:15:09.025215542 +0000 UTC m=+1855.999118081" Nov 01 22:15:09.315053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.314060 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 22:15:09.321055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.320144 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 22:15:09.321055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.320385 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" containerID="cri-o://ee503b0d030092d92344c309ca6a9d82db5ad4cc3d0883751e53703f16c39411" gracePeriod=30 Nov 01 22:15:09.326645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.326621 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 22:15:09.327145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.326876 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" containerID="cri-o://e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3" gracePeriod=30 Nov 01 22:15:09.416085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.415199 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:09.417744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.417694 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:09.417926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:09.417756 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:10.007349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:10.007309 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" containerID="cri-o://53d7bdd9c3b9a57f6a1c830beef6758880f5d4e8e3bc3467630baab029903d44" gracePeriod=30 Nov 01 22:15:10.450111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:10.448225 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:10.450111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:10.448286 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:11.405450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:11.405416 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:11.405826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:11.405469 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:12.405916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:12.405872 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:12.406417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:12.405942 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:13.406014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:13.405974 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:13.406487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:13.406089 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:14.431227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.410061 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Nov 01 22:15:14.431227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.410125 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Nov 01 22:15:14.614523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.614482 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b"] Nov 01 22:15:14.619181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.616137 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" containerName="revoke-keys" Nov 01 22:15:14.619181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.616158 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" containerName="revoke-keys" Nov 01 22:15:14.619181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.616299 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" containerName="revoke-keys" Nov 01 22:15:14.626053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.625775 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b"] Nov 01 22:15:14.626053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.625884 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.634927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.634875 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:15:14.708908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.708415 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72kr\" (UniqueName: \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.708908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.708507 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.708908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.708537 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.811712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.811493 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72kr\" (UniqueName: \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.811712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.811600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.811712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.811629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.812464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.812435 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.815086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.815060 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.825654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.825616 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72kr\" (UniqueName: \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\") pod \"maas-api-revoke-sub-5ddd222f-9nv5b\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:14.841879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.841848 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 22:15:14.948462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:14.948430 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:15.018667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018627 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.018841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018743 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.018841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018792 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.018957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018847 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.018957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018876 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.018957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.018919 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.019186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.019125 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.019325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.019307 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.019383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.019369 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\") pod \"9688198f-1431-474e-9ac7-92266000c8c4\" (UID: \"9688198f-1431-474e-9ac7-92266000c8c4\") " Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.020791 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.021416 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.021630 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.022283 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.022357 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.024104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.022754 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:15.028292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.026738 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.028292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.026840 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:15.060990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.060945 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.061217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.060989 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q" pod "9688198f-1431-474e-9ac7-92266000c8c4" (UID: "9688198f-1431-474e-9ac7-92266000c8c4"). InnerVolumeSpecName "kube-api-access-h2p9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.125205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125172 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9688198f-1431-474e-9ac7-92266000c8c4-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125205 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9688198f-1431-474e-9ac7-92266000c8c4-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125221 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125236 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125253 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125267 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125283 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2p9q\" (UniqueName: \"kubernetes.io/projected/9688198f-1431-474e-9ac7-92266000c8c4-kube-api-access-h2p9q\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.125663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.125296 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9688198f-1431-474e-9ac7-92266000c8c4-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.130387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.128482 4395 generic.go:356] "Generic (PLEG): container finished" podID="9688198f-1431-474e-9ac7-92266000c8c4" containerID="e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3" exitCode=0 Nov 01 22:15:15.130387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.128804 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" Nov 01 22:15:15.130387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.129248 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" event={"ID":"9688198f-1431-474e-9ac7-92266000c8c4","Type":"ContainerDied","Data":"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3"} Nov 01 22:15:15.130387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.129302 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk" event={"ID":"9688198f-1431-474e-9ac7-92266000c8c4","Type":"ContainerDied","Data":"bf67789fb7a64b6dd8fe70158f8bd4615fa35d2f5e3c4ede6d81bf0935914b93"} Nov 01 22:15:15.130387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.129343 4395 scope.go:122] "RemoveContainer" containerID="e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3" Nov 01 22:15:15.135428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.135397 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 22:15:15.138994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.138959 4395 generic.go:356] "Generic (PLEG): container finished" podID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerID="ee503b0d030092d92344c309ca6a9d82db5ad4cc3d0883751e53703f16c39411" exitCode=0 Nov 01 22:15:15.139313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.139282 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" event={"ID":"128a87b7-b7dd-44b2-9ac8-4271d835e526","Type":"ContainerDied","Data":"ee503b0d030092d92344c309ca6a9d82db5ad4cc3d0883751e53703f16c39411"} Nov 01 22:15:15.223431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.223114 4395 scope.go:122] "RemoveContainer" containerID="e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3" Nov 01 22:15:15.227217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.226979 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227249 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227279 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227406 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227438 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227478 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227550 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.227720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227607 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.228680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.227867 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\") pod \"128a87b7-b7dd-44b2-9ac8-4271d835e526\" (UID: \"128a87b7-b7dd-44b2-9ac8-4271d835e526\") " Nov 01 22:15:15.229109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.228840 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.229742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.229350 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:15.230182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.229884 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.235097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.231621 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:15.235097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.235050 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.235258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.235242 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.241278 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:15.237737 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3\": container with ID starting with e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3 not found: ID does not exist" containerID="e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3" Nov 01 22:15:15.241278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.237780 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3"} err="failed to get container status \"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3\": rpc error: code = NotFound desc = could not find container \"e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3\": container with ID starting with e5d3eed9d58102fb02642850a086f0bc3750a44398b1221f86cceb3e842892f3 not found: ID does not exist" Nov 01 22:15:15.241278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.237806 4395 scope.go:122] "RemoveContainer" containerID="ee503b0d030092d92344c309ca6a9d82db5ad4cc3d0883751e53703f16c39411" Nov 01 22:15:15.252094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.244112 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 22:15:15.252094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.251421 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.252094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.251547 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c8vgk"] Nov 01 22:15:15.256672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.253812 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.256672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.253850 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55" pod "128a87b7-b7dd-44b2-9ac8-4271d835e526" (UID: "128a87b7-b7dd-44b2-9ac8-4271d835e526"). InnerVolumeSpecName "kube-api-access-fxb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359900 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359935 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxb55\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-kube-api-access-fxb55\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359951 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359963 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359974 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359986 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.359997 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a87b7-b7dd-44b2-9ac8-4271d835e526-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.360008 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.403111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.360020 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a87b7-b7dd-44b2-9ac8-4271d835e526-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.444092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.441462 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b"] Nov 01 22:15:15.583473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.583438 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:15.585430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.585405 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:15.595684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.595539 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:15:15.695312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.694768 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 22:15:15.703384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.702315 4395 scope.go:122] "RemoveContainer" containerID="53d7bdd9c3b9a57f6a1c830beef6758880f5d4e8e3bc3467630baab029903d44" Nov 01 22:15:15.794575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.794544 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.794693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.794674 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.794753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.794715 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.794863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.794841 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.794930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.794901 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.795142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.795122 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.795232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.795150 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.795293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.795269 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.796467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.795394 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.796467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.795427 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\") pod \"7126f5d1-347c-4da5-a744-226fa0eb4e10\" (UID: \"7126f5d1-347c-4da5-a744-226fa0eb4e10\") " Nov 01 22:15:15.796467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.796119 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.796467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.796405 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.796467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.796425 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.796799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.796601 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.796799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.796740 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.797214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.797182 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:15.797939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.797473 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:15:15.802611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.801430 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.806530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.806501 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "kube-api-access-95r4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:15.807187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.807158 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo" pod "7126f5d1-347c-4da5-a744-226fa0eb4e10" (UID: "7126f5d1-347c-4da5-a744-226fa0eb4e10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:15:15.897711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897673 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897713 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95r4m\" (UniqueName: \"kubernetes.io/projected/7126f5d1-347c-4da5-a744-226fa0eb4e10-kube-api-access-95r4m\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897729 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897741 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7126f5d1-347c-4da5-a744-226fa0eb4e10-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897754 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897767 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:15.897936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:15.897777 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7126f5d1-347c-4da5-a744-226fa0eb4e10-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:16.159012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.158615 4395 scope.go:122] "RemoveContainer" containerID="db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d" Nov 01 22:15:16.168324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.159702 4395 scope.go:122] "RemoveContainer" containerID="d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f" Nov 01 22:15:16.168324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.160512 4395 scope.go:122] "RemoveContainer" containerID="a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69" Nov 01 22:15:16.191075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.185847 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" event={"ID":"128a87b7-b7dd-44b2-9ac8-4271d835e526","Type":"ContainerDied","Data":"bee77fc8ccf38e6d6f920189f5488d28dc46dc1230601244c1797979e9dac658"} Nov 01 22:15:16.191075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.185927 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" event={"ID":"7126f5d1-347c-4da5-a744-226fa0eb4e10","Type":"ContainerDied","Data":"53d7bdd9c3b9a57f6a1c830beef6758880f5d4e8e3bc3467630baab029903d44"} Nov 01 22:15:16.191075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.185958 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" event={"ID":"7126f5d1-347c-4da5-a744-226fa0eb4e10","Type":"ContainerDied","Data":"1df8f4f3b32ec176b3acf6269c0be7b9b12d5fb59cf7286d53c602a25550b27f"} Nov 01 22:15:16.242279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.242250 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt" Nov 01 22:15:16.244012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.243504 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5" Nov 01 22:15:16.245195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.242330 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" event={"ID":"aa611729-a77b-412b-9468-11557397eb9d","Type":"ContainerStarted","Data":"cb969808f0ed5ebc86981fc20ef9d063ddd7ccf928171e69992ebf26908f5a57"} Nov 01 22:15:16.245295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.245236 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" event={"ID":"aa611729-a77b-412b-9468-11557397eb9d","Type":"ContainerStarted","Data":"e9d5ad86271b5aab890615bfa24bc9124b7fbb20e637db419d5a115bda323201"} Nov 01 22:15:16.267560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.267480 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" podStartSLOduration=2.267464928 podStartE2EDuration="2.267464928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:16.263386709 +0000 UTC m=+1863.237289251" watchObservedRunningTime="2026-11-01 22:15:16.267464928 +0000 UTC m=+1863.241367467" Nov 01 22:15:16.335122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.335090 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 22:15:16.341563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.341534 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vwrt5"] Nov 01 22:15:16.352826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.352794 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 22:15:16.366296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:16.366265 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vktwt"] Nov 01 22:15:17.191134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.180141 4395 scope.go:122] "RemoveContainer" containerID="0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c" Nov 01 22:15:17.769382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.769313 4395 generic.go:356] "Generic (PLEG): container finished" podID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerID="39bab6073dd8a4f183f13cbc9c74f33f960c1415908b5055495267e341c2dbdb" exitCode=60 Nov 01 22:15:17.773081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.772987 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" path="/var/lib/kubelet/pods/128a87b7-b7dd-44b2-9ac8-4271d835e526/volumes" Nov 01 22:15:17.773977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.773953 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" path="/var/lib/kubelet/pods/7126f5d1-347c-4da5-a744-226fa0eb4e10/volumes" Nov 01 22:15:17.774768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.774736 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9688198f-1431-474e-9ac7-92266000c8c4" path="/var/lib/kubelet/pods/9688198f-1431-474e-9ac7-92266000c8c4/volumes" Nov 01 22:15:17.780347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.780312 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerDied","Data":"39bab6073dd8a4f183f13cbc9c74f33f960c1415908b5055495267e341c2dbdb"} Nov 01 22:15:17.780477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.780451 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:15:17.780560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.780542 4395 scope.go:122] "RemoveContainer" containerID="a73f9333aeef28ca03e6a68691994cc3917fe5ba7b8c0c17be1e0cb529ae4a69" Nov 01 22:15:17.780981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.780865 4395 generic.go:356] "Generic (PLEG): container finished" podID="aa611729-a77b-412b-9468-11557397eb9d" containerID="cb969808f0ed5ebc86981fc20ef9d063ddd7ccf928171e69992ebf26908f5a57" exitCode=0 Nov 01 22:15:17.782113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.782017 4395 scope.go:122] "RemoveContainer" containerID="39bab6073dd8a4f183f13cbc9c74f33f960c1415908b5055495267e341c2dbdb" Nov 01 22:15:17.782950 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:17.782924 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr_odh-ai-gateway-infra(908fe8ef-cb40-4b03-9957-56ef46f9d6aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" Nov 01 22:15:17.784192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784172 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" Nov 01 22:15:17.784316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784299 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" Nov 01 22:15:17.784484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784464 4395 cpu_manager.go:414] "containerMap: removing container" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" Nov 01 22:15:17.784597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784584 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" Nov 01 22:15:17.784779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784767 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" Nov 01 22:15:17.784859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.784849 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" Nov 01 22:15:17.786996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.786977 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9688198f-1431-474e-9ac7-92266000c8c4" containerName="istio-proxy" Nov 01 22:15:17.787215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.787191 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7126f5d1-347c-4da5-a744-226fa0eb4e10" containerName="istio-proxy" Nov 01 22:15:17.787372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.787358 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="128a87b7-b7dd-44b2-9ac8-4271d835e526" containerName="istio-proxy" Nov 01 22:15:17.811252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.811220 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:15:17.811498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.811475 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerStarted","Data":"91db17a3a91303dd878cc08d84704fb643f3a4bb1dd13bb7216ebbc9d1701f59"} Nov 01 22:15:17.811616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.811601 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" event={"ID":"aa611729-a77b-412b-9468-11557397eb9d","Type":"ContainerDied","Data":"cb969808f0ed5ebc86981fc20ef9d063ddd7ccf928171e69992ebf26908f5a57"} Nov 01 22:15:17.812802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.812778 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:17.820519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.815318 4395 generic.go:356] "Generic (PLEG): container finished" podID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerID="5e93c0142916430507c90c11d2d513e6fd8baab3c6ef68cca42f3dceb259ff33" exitCode=60 Nov 01 22:15:17.821598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.815347 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerDied","Data":"5e93c0142916430507c90c11d2d513e6fd8baab3c6ef68cca42f3dceb259ff33"} Nov 01 22:15:17.822152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.822133 4395 scope.go:122] "RemoveContainer" containerID="5e93c0142916430507c90c11d2d513e6fd8baab3c6ef68cca42f3dceb259ff33" Nov 01 22:15:17.822586 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:17.822562 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786_odh-ai-gateway-infra(a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" Nov 01 22:15:17.831251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.831228 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6b4c49-serving-cert\"" Nov 01 22:15:17.833213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.833196 4395 scope.go:122] "RemoveContainer" containerID="db23fa581285a7d533e24f3a2094e24e69c32bec1d2c7b889170bf9ae583b71d" Nov 01 22:15:17.918835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.918797 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:17.919577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.918893 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pr56\" (UniqueName: \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:17.919724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.919651 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:17.919785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:17.919737 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.022211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.022129 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.022211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.022216 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pr56\" (UniqueName: \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.022454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.022268 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.022454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.022373 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.023216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.023186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.024968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.024945 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.025077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.025059 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.034010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.033987 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pr56\" (UniqueName: \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\") pod \"maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.158324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.158298 4395 scope.go:122] "RemoveContainer" containerID="a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856" Nov 01 22:15:18.177656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.177628 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:18.542465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.542439 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:15:18.545652 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:18.545620 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e278919_5a21_46b3_a236_1c9fd18fb395.slice/crio-0342f34c45a9ce0bdfd37b9f1244fb42a1a6f8d90b1014ef82ee6c1c099386c0 WatchSource:0}: Error finding container 0342f34c45a9ce0bdfd37b9f1244fb42a1a6f8d90b1014ef82ee6c1c099386c0: Status 404 returned error can't find the container with id 0342f34c45a9ce0bdfd37b9f1244fb42a1a6f8d90b1014ef82ee6c1c099386c0 Nov 01 22:15:18.832692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.832606 4395 generic.go:356] "Generic (PLEG): container finished" podID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerID="91db17a3a91303dd878cc08d84704fb643f3a4bb1dd13bb7216ebbc9d1701f59" exitCode=60 Nov 01 22:15:18.832851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.832686 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerDied","Data":"91db17a3a91303dd878cc08d84704fb643f3a4bb1dd13bb7216ebbc9d1701f59"} Nov 01 22:15:18.832851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.832751 4395 scope.go:122] "RemoveContainer" containerID="d4ad3facdd529197ef48c189b8982e249e31bd98521d11760346913103f7a14f" Nov 01 22:15:18.833314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.833286 4395 scope.go:122] "RemoveContainer" containerID="91db17a3a91303dd878cc08d84704fb643f3a4bb1dd13bb7216ebbc9d1701f59" Nov 01 22:15:18.833797 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:18.833768 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4_odh-ai-gateway-infra(d26ee628-29b5-418f-9009-a9b75b08fc87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" Nov 01 22:15:18.835415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.835393 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerStarted","Data":"d920863a27602a7c6742501633f59a33530332a10275f3908058946d4efea1eb"} Nov 01 22:15:18.839909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.839887 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerStarted","Data":"42573f6bab7321341af89a9be7d9c09beb0a2295b09ec6dca9c6db3684b80833"} Nov 01 22:15:18.841623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.841599 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" event={"ID":"4e278919-5a21-46b3-a236-1c9fd18fb395","Type":"ContainerStarted","Data":"0342f34c45a9ce0bdfd37b9f1244fb42a1a6f8d90b1014ef82ee6c1c099386c0"} Nov 01 22:15:18.876063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.854803 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:18.947178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:18.943119 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:19.007286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.007257 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:19.190847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.186185 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:19.245704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.245673 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:19.271116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.271084 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:19.366181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366099 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\") pod \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " Nov 01 22:15:19.366357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366207 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\") pod \"aa611729-a77b-412b-9468-11557397eb9d\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " Nov 01 22:15:19.366357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366272 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\" (UniqueName: \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\") pod \"aa611729-a77b-412b-9468-11557397eb9d\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " Nov 01 22:15:19.366357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366323 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\") pod \"aa611729-a77b-412b-9468-11557397eb9d\" (UID: \"aa611729-a77b-412b-9468-11557397eb9d\") " Nov 01 22:15:19.366546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366407 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\" (UniqueName: \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\") pod \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\" (UID: \"908fe8ef-cb40-4b03-9957-56ef46f9d6aa\") " Nov 01 22:15:19.366546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366495 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\" (UniqueName: \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\") pod \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " Nov 01 22:15:19.366546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366498 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca" pod "908fe8ef-cb40-4b03-9957-56ef46f9d6aa" (UID: "908fe8ef-cb40-4b03-9957-56ef46f9d6aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:19.366713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366593 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\") pod \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\" (UID: \"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4\") " Nov 01 22:15:19.366850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.366829 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca" pod "aa611729-a77b-412b-9468-11557397eb9d" (UID: "aa611729-a77b-412b-9468-11557397eb9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:19.367483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.367457 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.367595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.367488 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa611729-a77b-412b-9468-11557397eb9d-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.369805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.369775 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h" pod "908fe8ef-cb40-4b03-9957-56ef46f9d6aa" (UID: "908fe8ef-cb40-4b03-9957-56ef46f9d6aa"). InnerVolumeSpecName "kube-api-access-6gz2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:19.370208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.370164 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca" pod "a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" (UID: "a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:19.370351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.370184 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr" pod "aa611729-a77b-412b-9468-11557397eb9d" (UID: "aa611729-a77b-412b-9468-11557397eb9d"). InnerVolumeSpecName "kube-api-access-q72kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:19.372409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.372389 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca" pod "aa611729-a77b-412b-9468-11557397eb9d" (UID: "aa611729-a77b-412b-9468-11557397eb9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:19.373857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.373836 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79" pod "a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" (UID: "a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4"). InnerVolumeSpecName "kube-api-access-n6v79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:19.468957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.468924 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa611729-a77b-412b-9468-11557397eb9d-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.468957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.468956 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72kr\" (UniqueName: \"kubernetes.io/projected/aa611729-a77b-412b-9468-11557397eb9d-kube-api-access-q72kr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.469206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.468967 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gz2h\" (UniqueName: \"kubernetes.io/projected/908fe8ef-cb40-4b03-9957-56ef46f9d6aa-kube-api-access-6gz2h\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.469206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.468976 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6v79\" (UniqueName: \"kubernetes.io/projected/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-kube-api-access-n6v79\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.469206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.468984 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:19.857819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.857757 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" event={"ID":"4e278919-5a21-46b3-a236-1c9fd18fb395","Type":"ContainerStarted","Data":"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028"} Nov 01 22:15:19.858964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.858939 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:19.861346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.861323 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" Nov 01 22:15:19.861482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.861347 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr" event={"ID":"908fe8ef-cb40-4b03-9957-56ef46f9d6aa","Type":"ContainerDied","Data":"1db68d971f32da8cc19789ab34d5dcbf01bf4a38d71227f915f8cae71d93f866"} Nov 01 22:15:19.861482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.861403 4395 scope.go:122] "RemoveContainer" containerID="39bab6073dd8a4f183f13cbc9c74f33f960c1415908b5055495267e341c2dbdb" Nov 01 22:15:19.868843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.868817 4395 generic.go:356] "Generic (PLEG): container finished" podID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerID="d920863a27602a7c6742501633f59a33530332a10275f3908058946d4efea1eb" exitCode=60 Nov 01 22:15:19.868992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.868904 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerDied","Data":"d920863a27602a7c6742501633f59a33530332a10275f3908058946d4efea1eb"} Nov 01 22:15:19.869408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.869387 4395 scope.go:122] "RemoveContainer" containerID="d920863a27602a7c6742501633f59a33530332a10275f3908058946d4efea1eb" Nov 01 22:15:19.870511 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:19.869796 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x_odh-ai-gateway-infra(e8bee970-09ad-4a69-b3e9-79c68b3313f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" Nov 01 22:15:19.875941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.871890 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" event={"ID":"aa611729-a77b-412b-9468-11557397eb9d","Type":"ContainerDied","Data":"e9d5ad86271b5aab890615bfa24bc9124b7fbb20e637db419d5a115bda323201"} Nov 01 22:15:19.875941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.871947 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d5ad86271b5aab890615bfa24bc9124b7fbb20e637db419d5a115bda323201" Nov 01 22:15:19.875941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.872009 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b" Nov 01 22:15:19.887640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.885800 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" podStartSLOduration=2.455284577 podStartE2EDuration="2.885784017s" totalImagesPullingTime="430.49944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:19.881689316 +0000 UTC m=+1866.855591867" watchObservedRunningTime="2026-11-01 22:15:19.885784017 +0000 UTC m=+1866.859686559" Nov 01 22:15:19.889081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.889022 4395 scope.go:122] "RemoveContainer" containerID="a4551b5282ac0a7ad84e1531390bfaedd9d45576c2a4f70912a7907ca986f856" Nov 01 22:15:19.894696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.894639 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" event={"ID":"a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4","Type":"ContainerDied","Data":"2d08ecbee5b2b5a34983e72c19c609606f76f993d050a85d3cb799534e7a39a7"} Nov 01 22:15:19.894836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.894812 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786" Nov 01 22:15:19.910402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.910378 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d4211a2-1465-497f-9165-70f935c78220" containerID="42573f6bab7321341af89a9be7d9c09beb0a2295b09ec6dca9c6db3684b80833" exitCode=60 Nov 01 22:15:19.912182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.912153 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerDied","Data":"42573f6bab7321341af89a9be7d9c09beb0a2295b09ec6dca9c6db3684b80833"} Nov 01 22:15:19.912304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.912276 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:19.912670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.912647 4395 scope.go:122] "RemoveContainer" containerID="42573f6bab7321341af89a9be7d9c09beb0a2295b09ec6dca9c6db3684b80833" Nov 01 22:15:19.913309 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:19.913281 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn_odh-ai-gateway-infra(6d4211a2-1465-497f-9165-70f935c78220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" podUID="6d4211a2-1465-497f-9165-70f935c78220" Nov 01 22:15:19.938072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.937968 4395 scope.go:122] "RemoveContainer" containerID="5e93c0142916430507c90c11d2d513e6fd8baab3c6ef68cca42f3dceb259ff33" Nov 01 22:15:19.953497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.953470 4395 scope.go:122] "RemoveContainer" containerID="0edd9dfefa2a1e0ec4f9aab193278c04e4fe055e8ac6ffe3b6fae47df175bf7c" Nov 01 22:15:19.954448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.954409 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:19.988138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.986482 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:19.993634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:19.993603 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892855-8s786"] Nov 01 22:15:20.259722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.250471 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:20.259722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.259371 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f3a6b-29892855-jfxvr"] Nov 01 22:15:20.385393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.385363 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:20.452344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.452305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\" (UniqueName: \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\") pod \"d26ee628-29b5-418f-9009-a9b75b08fc87\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " Nov 01 22:15:20.452546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.452470 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\") pod \"d26ee628-29b5-418f-9009-a9b75b08fc87\" (UID: \"d26ee628-29b5-418f-9009-a9b75b08fc87\") " Nov 01 22:15:20.453623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.453584 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca" pod "d26ee628-29b5-418f-9009-a9b75b08fc87" (UID: "d26ee628-29b5-418f-9009-a9b75b08fc87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:20.465856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.465821 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7" pod "d26ee628-29b5-418f-9009-a9b75b08fc87" (UID: "d26ee628-29b5-418f-9009-a9b75b08fc87"). InnerVolumeSpecName "kube-api-access-mxjh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:20.555567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.555489 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ee628-29b5-418f-9009-a9b75b08fc87-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:20.555567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.555518 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjh7\" (UniqueName: \"kubernetes.io/projected/d26ee628-29b5-418f-9009-a9b75b08fc87-kube-api-access-mxjh7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:20.601738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.601701 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:15:20.603090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603064 4395 cpu_manager.go:414] "containerMap: removing container" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.603090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603089 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603109 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603118 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603127 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603136 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603151 4395 cpu_manager.go:414] "containerMap: removing container" podUID="aa611729-a77b-412b-9468-11557397eb9d" containerName="revoke-keys" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603159 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa611729-a77b-412b-9468-11557397eb9d" containerName="revoke-keys" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603172 4395 cpu_manager.go:414] "containerMap: removing container" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603181 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603222 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603231 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603263 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.603359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603271 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603526 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603548 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603563 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603573 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603586 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa611729-a77b-412b-9468-11557397eb9d" containerName="revoke-keys" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603596 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:20.604103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.603608 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:20.619783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.619753 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.649079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.649054 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:15:20.674698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.674674 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-da34ca-openshift-default-dockercfg-2gmfx\"" Nov 01 22:15:20.784059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.761580 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.784501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.784467 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.784965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.784942 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.785188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.785170 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.785557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.785536 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.785751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.785734 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.785920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.785906 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.786098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.786081 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqw2d\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.786518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.786494 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.894230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.894142 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.895716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.895691 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.895927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.895912 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.896055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.896020 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqw2d\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.896325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.896308 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.896524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.896512 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.905049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.896664 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.905326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.905293 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.905483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.905465 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.906395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.895629 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.907003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.906979 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.907940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.907918 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.909220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.908551 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.909342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.909167 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.923691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.923665 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.956540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.956505 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.982115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.968735 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqw2d\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:20.982115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:20.971673 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:21.029045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.025405 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" event={"ID":"d26ee628-29b5-418f-9009-a9b75b08fc87","Type":"ContainerDied","Data":"9013bbbd06d0ab680364ce7485f7b1ab41dc53657ab97e816cfa21c5a2a75bbd"} Nov 01 22:15:21.029045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.025477 4395 scope.go:122] "RemoveContainer" containerID="91db17a3a91303dd878cc08d84704fb643f3a4bb1dd13bb7216ebbc9d1701f59" Nov 01 22:15:21.029045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.025589 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4" Nov 01 22:15:21.123831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.123799 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:21.129913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.129208 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1783e3-29892855-6k5j4"] Nov 01 22:15:21.254414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.254140 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:21.289198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.289143 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" path="/var/lib/kubelet/pods/908fe8ef-cb40-4b03-9957-56ef46f9d6aa/volumes" Nov 01 22:15:21.298290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.298257 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" path="/var/lib/kubelet/pods/a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4/volumes" Nov 01 22:15:21.299338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.299310 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" path="/var/lib/kubelet/pods/d26ee628-29b5-418f-9009-a9b75b08fc87/volumes" Nov 01 22:15:21.431509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.431467 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:21.483679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.483655 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:21.565593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.565547 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\") pod \"6d4211a2-1465-497f-9165-70f935c78220\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " Nov 01 22:15:21.571926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.571894 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\") pod \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " Nov 01 22:15:21.572217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.572199 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\" (UniqueName: \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\") pod \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\" (UID: \"e8bee970-09ad-4a69-b3e9-79c68b3313f8\") " Nov 01 22:15:21.572383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.572365 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\" (UniqueName: \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\") pod \"6d4211a2-1465-497f-9165-70f935c78220\" (UID: \"6d4211a2-1465-497f-9165-70f935c78220\") " Nov 01 22:15:21.573563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.566291 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca" pod "6d4211a2-1465-497f-9165-70f935c78220" (UID: "6d4211a2-1465-497f-9165-70f935c78220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:21.573563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.573232 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca" pod "e8bee970-09ad-4a69-b3e9-79c68b3313f8" (UID: "e8bee970-09ad-4a69-b3e9-79c68b3313f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:21.573773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.573651 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d4211a2-1465-497f-9165-70f935c78220-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:21.573773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.573680 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8bee970-09ad-4a69-b3e9-79c68b3313f8-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:21.637848 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.637791 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5" pod "e8bee970-09ad-4a69-b3e9-79c68b3313f8" (UID: "e8bee970-09ad-4a69-b3e9-79c68b3313f8"). InnerVolumeSpecName "kube-api-access-hmhs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:21.639124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.639082 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p" pod "6d4211a2-1465-497f-9165-70f935c78220" (UID: "6d4211a2-1465-497f-9165-70f935c78220"). InnerVolumeSpecName "kube-api-access-6jc7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:21.662561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.662508 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:15:21.669462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.669402 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:21.669629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.669588 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:21.670235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.669699 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:15:21.677083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.677008 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhs5\" (UniqueName: \"kubernetes.io/projected/e8bee970-09ad-4a69-b3e9-79c68b3313f8-kube-api-access-hmhs5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:21.677083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:21.677056 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jc7p\" (UniqueName: \"kubernetes.io/projected/6d4211a2-1465-497f-9165-70f935c78220-kube-api-access-6jc7p\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:22.050514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.050484 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" Nov 01 22:15:22.051647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.051059 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x" event={"ID":"e8bee970-09ad-4a69-b3e9-79c68b3313f8","Type":"ContainerDied","Data":"39b46438b974e1f260eca50efd2966722c2a691d6a1b3bf06ffae02d2af53b49"} Nov 01 22:15:22.052162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.052145 4395 scope.go:122] "RemoveContainer" containerID="d920863a27602a7c6742501633f59a33530332a10275f3908058946d4efea1eb" Nov 01 22:15:22.053732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.053703 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" event={"ID":"6d4211a2-1465-497f-9165-70f935c78220","Type":"ContainerDied","Data":"027b67051402d8934dda4478a9bb96b861fe6c8640fa1ad7d530b7800c08986a"} Nov 01 22:15:22.053901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.053811 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn" Nov 01 22:15:22.058679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.058650 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerStarted","Data":"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad"} Nov 01 22:15:22.058782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.058700 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerStarted","Data":"1c0a0d647c15373e06459f5c685b72bcde32e23a398522ceb55a8050134129fe"} Nov 01 22:15:22.169626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.169553 4395 scope.go:122] "RemoveContainer" containerID="42573f6bab7321341af89a9be7d9c09beb0a2295b09ec6dca9c6db3684b80833" Nov 01 22:15:22.186351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.186219 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podStartSLOduration=2.186195672 podStartE2EDuration="2.186195672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:22.160430116 +0000 UTC m=+1869.134332680" watchObservedRunningTime="2026-11-01 22:15:22.186195672 +0000 UTC m=+1869.160098215" Nov 01 22:15:22.275062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.268677 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:22.281849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.281805 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:15:22.282185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.282152 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:15:22.306617 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.306526 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:22.365565 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:22.365530 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:15:22.371353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.371292 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892855-j4bpn"] Nov 01 22:15:22.379597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.379529 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:22.398476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:22.398446 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-2c5135-29892855-x9x6x"] Nov 01 22:15:23.168798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:23.168766 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4211a2-1465-497f-9165-70f935c78220" path="/var/lib/kubelet/pods/6d4211a2-1465-497f-9165-70f935c78220/volumes" Nov 01 22:15:23.169257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:23.169244 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" path="/var/lib/kubelet/pods/e8bee970-09ad-4a69-b3e9-79c68b3313f8/volumes" Nov 01 22:15:23.262266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:23.262238 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:24.090317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:24.090288 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:24.091548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:24.091524 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:15:25.693638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.693600 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:15:25.694542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694523 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694545 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694572 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694578 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694586 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694592 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:25.694612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694610 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694615 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694623 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694628 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694637 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694641 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694659 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694664 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694672 4395 cpu_manager.go:414] "containerMap: removing container" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694676 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694793 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694804 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694811 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694818 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bee970-09ad-4a69-b3e9-79c68b3313f8" containerName="cleanup" Nov 01 22:15:25.694823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694826 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ee628-29b5-418f-9009-a9b75b08fc87" containerName="cleanup" Nov 01 22:15:25.695257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694833 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.695257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694840 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e0c015-1ebc-4d45-b409-0aadc0b1b7b4" containerName="cleanup" Nov 01 22:15:25.695257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694846 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="908fe8ef-cb40-4b03-9957-56ef46f9d6aa" containerName="cleanup" Nov 01 22:15:25.695257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.694854 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.700394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.700370 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.707182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.707157 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-da34ca\"" Nov 01 22:15:25.710967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.710938 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:15:25.716640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.716615 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:15:25.717876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.717855 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.718050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.717877 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4211a2-1465-497f-9165-70f935c78220" containerName="cleanup" Nov 01 22:15:25.724943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.724920 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.728904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.728878 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-da34ca-dockercfg-hhkkq\"" Nov 01 22:15:25.740892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.740866 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:15:25.873068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.840583 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwx5\" (UniqueName: \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.873068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.840831 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vxs\" (UniqueName: \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.873068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.840876 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.873068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.840912 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.944625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.944520 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwx5\" (UniqueName: \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.944999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.944973 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vxs\" (UniqueName: \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.945223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.945201 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.945364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.945346 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.949063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.946321 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:25.949063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.947489 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.982009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.981976 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwx5\" (UniqueName: \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\") pod \"payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:25.995735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:25.995702 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vxs\" (UniqueName: \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\") pod \"payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:26.038782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:26.038480 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:26.066071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:26.056163 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:26.505834 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:26.505803 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f491e2_7944_449a_bc2a_25d212be6c57.slice/crio-36a91ca7b854ee2c99bbeebdb3be5d77435873b0db6b5e0308df650e4154be00 WatchSource:0}: Error finding container 36a91ca7b854ee2c99bbeebdb3be5d77435873b0db6b5e0308df650e4154be00: Status 404 returned error can't find the container with id 36a91ca7b854ee2c99bbeebdb3be5d77435873b0db6b5e0308df650e4154be00 Nov 01 22:15:26.506969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:26.506938 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:15:26.556097 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:15:26.556042 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63208df_f792_41e8_9f8a_cd459084c2d6.slice/crio-1668e9f5a2edb798d7b406830686028812765254d11b579513b95ac8b0792b83 WatchSource:0}: Error finding container 1668e9f5a2edb798d7b406830686028812765254d11b579513b95ac8b0792b83: Status 404 returned error can't find the container with id 1668e9f5a2edb798d7b406830686028812765254d11b579513b95ac8b0792b83 Nov 01 22:15:26.558020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:26.557963 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:15:27.063850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.063823 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:15:27.236383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.236341 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" event={"ID":"e63208df-f792-41e8-9f8a-cd459084c2d6","Type":"ContainerStarted","Data":"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418"} Nov 01 22:15:27.236572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.236412 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" event={"ID":"e63208df-f792-41e8-9f8a-cd459084c2d6","Type":"ContainerStarted","Data":"1668e9f5a2edb798d7b406830686028812765254d11b579513b95ac8b0792b83"} Nov 01 22:15:27.236572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.236446 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:27.238157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.238131 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" event={"ID":"09f491e2-7944-449a-bc2a-25d212be6c57","Type":"ContainerStarted","Data":"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465"} Nov 01 22:15:27.238316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.238182 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" event={"ID":"09f491e2-7944-449a-bc2a-25d212be6c57","Type":"ContainerStarted","Data":"36a91ca7b854ee2c99bbeebdb3be5d77435873b0db6b5e0308df650e4154be00"} Nov 01 22:15:27.238316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.238212 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:27.256198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.256157 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" podStartSLOduration=2.25614251 podStartE2EDuration="2.25614251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:27.254212472 +0000 UTC m=+1874.228115011" watchObservedRunningTime="2026-11-01 22:15:27.25614251 +0000 UTC m=+1874.230045050" Nov 01 22:15:27.276744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:27.276695 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" podStartSLOduration=2.276682562 podStartE2EDuration="2.276682562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:27.271337714 +0000 UTC m=+1874.245240255" watchObservedRunningTime="2026-11-01 22:15:27.276682562 +0000 UTC m=+1874.250585126" Nov 01 22:15:38.249158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:38.249130 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:15:38.249553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:38.249256 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:15:43.969800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:43.969761 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:15:46.106060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.106008 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:15:46.106879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.106847 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" podUID="4386dc80-d310-4fcd-8956-5085a17c1f2f" containerName="maas-api" containerID="cri-o://ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249" gracePeriod=30 Nov 01 22:15:46.237134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.237062 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:15:46.237737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.237683 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" podUID="44424331-241a-4818-aecd-b429fec79810" containerName="maas-api" containerID="cri-o://86a157db20ceed175a7bc1ed10a2efd2aacc83036b79fc384532ca6c07cca3ba" gracePeriod=30 Nov 01 22:15:46.606050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.603227 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:15:46.651086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.648631 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\") pod \"4386dc80-d310-4fcd-8956-5085a17c1f2f\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " Nov 01 22:15:46.651086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.648690 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\" (UniqueName: \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\") pod \"4386dc80-d310-4fcd-8956-5085a17c1f2f\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " Nov 01 22:15:46.651086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.648837 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\") pod \"4386dc80-d310-4fcd-8956-5085a17c1f2f\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " Nov 01 22:15:46.651086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.648889 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") pod \"4386dc80-d310-4fcd-8956-5085a17c1f2f\" (UID: \"4386dc80-d310-4fcd-8956-5085a17c1f2f\") " Nov 01 22:15:46.674071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.672318 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca" pod "4386dc80-d310-4fcd-8956-5085a17c1f2f" (UID: "4386dc80-d310-4fcd-8956-5085a17c1f2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:46.759622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.755780 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4386dc80-d310-4fcd-8956-5085a17c1f2f-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:46.816176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.816094 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs" pod "4386dc80-d310-4fcd-8956-5085a17c1f2f" (UID: "4386dc80-d310-4fcd-8956-5085a17c1f2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:46.828271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.828223 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf" pod "4386dc80-d310-4fcd-8956-5085a17c1f2f" (UID: "4386dc80-d310-4fcd-8956-5085a17c1f2f"). InnerVolumeSpecName "kube-api-access-b8fpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:46.851359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.851319 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls" pod "4386dc80-d310-4fcd-8956-5085a17c1f2f" (UID: "4386dc80-d310-4fcd-8956-5085a17c1f2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:46.862172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.859846 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:46.862172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.859880 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8fpf\" (UniqueName: \"kubernetes.io/projected/4386dc80-d310-4fcd-8956-5085a17c1f2f-kube-api-access-b8fpf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:46.862172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.859901 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4386dc80-d310-4fcd-8956-5085a17c1f2f-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:46.895992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.895963 4395 generic.go:356] "Generic (PLEG): container finished" podID="44424331-241a-4818-aecd-b429fec79810" containerID="86a157db20ceed175a7bc1ed10a2efd2aacc83036b79fc384532ca6c07cca3ba" exitCode=0 Nov 01 22:15:46.896434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.896104 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" event={"ID":"44424331-241a-4818-aecd-b429fec79810","Type":"ContainerDied","Data":"86a157db20ceed175a7bc1ed10a2efd2aacc83036b79fc384532ca6c07cca3ba"} Nov 01 22:15:46.903972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.903937 4395 generic.go:356] "Generic (PLEG): container finished" podID="4386dc80-d310-4fcd-8956-5085a17c1f2f" containerID="ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249" exitCode=0 Nov 01 22:15:46.908128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.904138 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" event={"ID":"4386dc80-d310-4fcd-8956-5085a17c1f2f","Type":"ContainerDied","Data":"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249"} Nov 01 22:15:46.908368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.908340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" event={"ID":"4386dc80-d310-4fcd-8956-5085a17c1f2f","Type":"ContainerDied","Data":"c543d192e840b39c58131f7e29afd14b512f0f7dc4f4e2ea033bbded995dc08f"} Nov 01 22:15:46.908451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.908395 4395 scope.go:122] "RemoveContainer" containerID="ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249" Nov 01 22:15:46.908522 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.904242 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9" Nov 01 22:15:46.960866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.960822 4395 scope.go:122] "RemoveContainer" containerID="ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249" Nov 01 22:15:46.961339 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:46.961304 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249\": container with ID starting with ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249 not found: ID does not exist" containerID="ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249" Nov 01 22:15:46.961434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.961343 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:15:46.961434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:46.961353 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249"} err="failed to get container status \"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249\": rpc error: code = NotFound desc = could not find container \"ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249\": container with ID starting with ed19feb92181de31719992d553e3d9e3e3ca384f39bee01822aeb362f8136249 not found: ID does not exist" Nov 01 22:15:47.010083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.009968 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:15:47.019100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.018991 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0a5de-6f86ff5589-qcvq9"] Nov 01 22:15:47.061827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.061790 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\" (UniqueName: \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\") pod \"44424331-241a-4818-aecd-b429fec79810\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " Nov 01 22:15:47.062176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.062084 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\") pod \"44424331-241a-4818-aecd-b429fec79810\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " Nov 01 22:15:47.062330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.062271 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\") pod \"44424331-241a-4818-aecd-b429fec79810\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " Nov 01 22:15:47.062392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.062340 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\") pod \"44424331-241a-4818-aecd-b429fec79810\" (UID: \"44424331-241a-4818-aecd-b429fec79810\") " Nov 01 22:15:47.063723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.063589 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca" pod "44424331-241a-4818-aecd-b429fec79810" (UID: "44424331-241a-4818-aecd-b429fec79810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:15:47.071050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.070965 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs" pod "44424331-241a-4818-aecd-b429fec79810" (UID: "44424331-241a-4818-aecd-b429fec79810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:47.071050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.071002 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls" pod "44424331-241a-4818-aecd-b429fec79810" (UID: "44424331-241a-4818-aecd-b429fec79810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:15:47.071208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.071098 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j" pod "44424331-241a-4818-aecd-b429fec79810" (UID: "44424331-241a-4818-aecd-b429fec79810"). InnerVolumeSpecName "kube-api-access-rfz4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:15:47.080710 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:47.080690 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4386dc80_d310_4fcd_8956_5085a17c1f2f.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:15:47.163917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.163887 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44424331-241a-4818-aecd-b429fec79810-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:47.163917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.163917 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:47.164319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.163927 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44424331-241a-4818-aecd-b429fec79810-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:47.164319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.163936 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfz4j\" (UniqueName: \"kubernetes.io/projected/44424331-241a-4818-aecd-b429fec79810-kube-api-access-rfz4j\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:15:47.168390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.168366 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4386dc80-d310-4fcd-8956-5085a17c1f2f" path="/var/lib/kubelet/pods/4386dc80-d310-4fcd-8956-5085a17c1f2f/volumes" Nov 01 22:15:47.919795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.919762 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" event={"ID":"44424331-241a-4818-aecd-b429fec79810","Type":"ContainerDied","Data":"0be004cff457b498cb706357099c19108229af9c2da6029715d7c26eeab01d05"} Nov 01 22:15:47.919974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.919802 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d" Nov 01 22:15:47.919974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.919827 4395 scope.go:122] "RemoveContainer" containerID="86a157db20ceed175a7bc1ed10a2efd2aacc83036b79fc384532ca6c07cca3ba" Nov 01 22:15:47.942069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.942021 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:15:47.946650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.946627 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f2760d-79fc549865-fjd2d"] Nov 01 22:15:47.992876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.992842 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:15:47.993800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.993776 4395 cpu_manager.go:414] "containerMap: removing container" podUID="44424331-241a-4818-aecd-b429fec79810" containerName="maas-api" Nov 01 22:15:47.993800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.993797 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44424331-241a-4818-aecd-b429fec79810" containerName="maas-api" Nov 01 22:15:47.993987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.993861 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4386dc80-d310-4fcd-8956-5085a17c1f2f" containerName="maas-api" Nov 01 22:15:47.993987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.993867 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386dc80-d310-4fcd-8956-5085a17c1f2f" containerName="maas-api" Nov 01 22:15:47.994204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.994130 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386dc80-d310-4fcd-8956-5085a17c1f2f" containerName="maas-api" Nov 01 22:15:47.994204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:47.994156 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="44424331-241a-4818-aecd-b429fec79810" containerName="maas-api" Nov 01 22:15:48.001242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.001218 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.008338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.008311 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:15:48.014191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.013760 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-da34ca-serving-cert\"" Nov 01 22:15:48.080788 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.080761 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.080911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.080819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.080973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.080953 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhh6\" (UniqueName: \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.081149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.081131 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.194134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.193669 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.194134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.193747 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.194134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.193811 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhh6\" (UniqueName: \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.221056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.209011 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.221056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.193997 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.242837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.239775 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhh6\" (UniqueName: \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.242837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.242388 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.246565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.246535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.320611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.320585 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:48.749063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.747166 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:15:48.929551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.929514 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" event={"ID":"cb175887-0bf0-4c55-a1af-f5492f3a9506","Type":"ContainerStarted","Data":"2ccb13a60b58828aebccf6893a570478567a505b0c698b03f744cace26273789"} Nov 01 22:15:48.970811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:48.970761 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" containerID="cri-o://4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43" gracePeriod=25 Nov 01 22:15:49.174243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:49.174204 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44424331-241a-4818-aecd-b429fec79810" path="/var/lib/kubelet/pods/44424331-241a-4818-aecd-b429fec79810/volumes" Nov 01 22:15:49.945208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:49.945160 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" event={"ID":"cb175887-0bf0-4c55-a1af-f5492f3a9506","Type":"ContainerStarted","Data":"23ace1ff377a2099a11588820cab0f57e8abd41642934b454a82996749968eb2"} Nov 01 22:15:49.945688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:49.945257 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:50.776812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:50.776774 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:15:52.459057 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:15:52.455363 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:15:55.960529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:55.960501 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:15:55.987825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:55.987762 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" podStartSLOduration=8.478896099 podStartE2EDuration="8.987743909s" totalImagesPullingTime="508.84781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:49.965852545 +0000 UTC m=+1896.939755077" watchObservedRunningTime="2026-11-01 22:15:55.987743909 +0000 UTC m=+1902.961646449" Nov 01 22:15:59.969971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:15:59.969934 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:16:00.775826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:00.775790 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:16:04.978064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:04.971181 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" containerID="cri-o://57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9" gracePeriod=25 Nov 01 22:16:05.029325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.029290 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:16:05.400908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.400870 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:16:05.435330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.435294 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.540080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.539655 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dfdb11-openshift-default-dockercfg-8h45l\"" Nov 01 22:16:05.605892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.605839 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.606465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.606412 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.606660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.606642 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.607003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.606971 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.607332 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.607291 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.607689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.607669 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5cj\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.607906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.607888 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.608181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.608161 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.608554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.608535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.616755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.616699 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711086 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711148 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711180 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711351 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711390 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711413 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5cj\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711421 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711442 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.711482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711476 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.712102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711558 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.712102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.711591 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.712102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.712020 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.712728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.712703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.713213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.713152 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.714626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.714601 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.715116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.715096 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.725272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.725247 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.725387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.725325 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5cj\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\") pod \"e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:05.974995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:05.974909 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:06.474526 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:16:06.474498 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698859d6_5084_4b99_8d80_610441e98fea.slice/crio-8031237a33ca6961c1225abf72243ad960d0ea0c97207587e66486b68db6069b WatchSource:0}: Error finding container 8031237a33ca6961c1225abf72243ad960d0ea0c97207587e66486b68db6069b: Status 404 returned error can't find the container with id 8031237a33ca6961c1225abf72243ad960d0ea0c97207587e66486b68db6069b Nov 01 22:16:06.477777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:06.477746 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:16:06.477866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:06.477814 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:16:06.477866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:06.477843 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:16:06.477935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:06.477876 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:16:07.443211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.443171 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" event={"ID":"698859d6-5084-4b99-8d80-610441e98fea","Type":"ContainerStarted","Data":"19d01e68bef4624cc77c2ce739439dece113385f7512359e28c90285fcfdf13b"} Nov 01 22:16:07.443405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.443236 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" event={"ID":"698859d6-5084-4b99-8d80-610441e98fea","Type":"ContainerStarted","Data":"8031237a33ca6961c1225abf72243ad960d0ea0c97207587e66486b68db6069b"} Nov 01 22:16:07.471964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.471908 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podStartSLOduration=2.471891123 podStartE2EDuration="2.471891123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:07.467370499 +0000 UTC m=+1914.441273039" watchObservedRunningTime="2026-11-01 22:16:07.471891123 +0000 UTC m=+1914.445793664" Nov 01 22:16:07.999076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.997749 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:07.999859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.999822 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:07.999978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:07.999889 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:08.995060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:08.980240 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:08.995060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:08.980299 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:09.346298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.346273 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:16:09.458274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.455010 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\" (UniqueName: \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\") pod \"a2c54621-69e3-4946-9ef9-ccf05fff0199\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " Nov 01 22:16:09.458274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.455107 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\") pod \"a2c54621-69e3-4946-9ef9-ccf05fff0199\" (UID: \"a2c54621-69e3-4946-9ef9-ccf05fff0199\") " Nov 01 22:16:09.458274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.456706 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume" pod "a2c54621-69e3-4946-9ef9-ccf05fff0199" (UID: "a2c54621-69e3-4946-9ef9-ccf05fff0199"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:09.473702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.473662 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd" pod "a2c54621-69e3-4946-9ef9-ccf05fff0199" (UID: "a2c54621-69e3-4946-9ef9-ccf05fff0199"). InnerVolumeSpecName "kube-api-access-gdlxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:09.527353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.527316 4395 generic.go:356] "Generic (PLEG): container finished" podID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerID="4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43" exitCode=0 Nov 01 22:16:09.532180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.532149 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" event={"ID":"a2c54621-69e3-4946-9ef9-ccf05fff0199","Type":"ContainerDied","Data":"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43"} Nov 01 22:16:09.532414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.532393 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" event={"ID":"a2c54621-69e3-4946-9ef9-ccf05fff0199","Type":"ContainerDied","Data":"e8484751bbafade7e252e46899977352411ffe7500d55ddaf3790a12b513ebca"} Nov 01 22:16:09.532538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.532524 4395 scope.go:122] "RemoveContainer" containerID="4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43" Nov 01 22:16:09.532760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.532746 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw" Nov 01 22:16:09.549539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.549520 4395 scope.go:122] "RemoveContainer" containerID="4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43" Nov 01 22:16:09.549801 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:09.549783 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43\": container with ID starting with 4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43 not found: ID does not exist" containerID="4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43" Nov 01 22:16:09.549837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.549808 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43"} err="failed to get container status \"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43\": rpc error: code = NotFound desc = could not find container \"4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43\": container with ID starting with 4ff246507924ac1746dc8a541fcca1609f1b5210b82f4086fc18b90a8ddb2c43 not found: ID does not exist" Nov 01 22:16:09.558433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.558404 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlxd\" (UniqueName: \"kubernetes.io/projected/a2c54621-69e3-4946-9ef9-ccf05fff0199-kube-api-access-gdlxd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:09.558433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.558433 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c54621-69e3-4946-9ef9-ccf05fff0199-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:09.560931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.560853 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:16:09.565625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.565606 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f2760d-d96bc568c-9grkw"] Nov 01 22:16:10.000119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.998431 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:10.000119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:09.998489 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:10.030304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:10.030259 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" containerID="cri-o://4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276" gracePeriod=25 Nov 01 22:16:10.970233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:10.970144 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:16:11.004408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:11.004367 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:11.004605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:11.004433 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:11.238093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:11.237937 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" path="/var/lib/kubelet/pods/a2c54621-69e3-4946-9ef9-ccf05fff0199/volumes" Nov 01 22:16:11.985064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:11.977451 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:11.985064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:11.977537 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:12.986054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:12.978401 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:16:12.986054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:12.978482 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:16:14.029902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:14.023416 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:14.029902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:14.026588 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:14.033938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:14.033283 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:16:19.085265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.085232 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:16:19.085827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.085590 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" podUID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" containerName="maas-api" containerID="cri-o://edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304" gracePeriod=30 Nov 01 22:16:19.397864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.397508 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:16:19.453465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.453442 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:16:19.497553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.497526 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\") pod \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " Nov 01 22:16:19.497686 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.497623 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\") pod \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " Nov 01 22:16:19.497726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.497699 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\" (UniqueName: \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\") pod \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " Nov 01 22:16:19.497812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.497797 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\") pod \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\" (UID: \"2db0c166-47aa-4aa8-be5d-f949cbddfbd7\") " Nov 01 22:16:19.498194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.498169 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca" pod "2db0c166-47aa-4aa8-be5d-f949cbddfbd7" (UID: "2db0c166-47aa-4aa8-be5d-f949cbddfbd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:19.498664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.498648 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:19.500106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.500086 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf" pod "2db0c166-47aa-4aa8-be5d-f949cbddfbd7" (UID: "2db0c166-47aa-4aa8-be5d-f949cbddfbd7"). InnerVolumeSpecName "kube-api-access-wzrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:19.500377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.500356 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs" pod "2db0c166-47aa-4aa8-be5d-f949cbddfbd7" (UID: "2db0c166-47aa-4aa8-be5d-f949cbddfbd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:19.500439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.500384 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls" pod "2db0c166-47aa-4aa8-be5d-f949cbddfbd7" (UID: "2db0c166-47aa-4aa8-be5d-f949cbddfbd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:19.600960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.600917 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:19.600960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.600954 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:19.601271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.600972 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrhf\" (UniqueName: \"kubernetes.io/projected/2db0c166-47aa-4aa8-be5d-f949cbddfbd7-kube-api-access-wzrhf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:19.719931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.719842 4395 generic.go:356] "Generic (PLEG): container finished" podID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" containerID="edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304" exitCode=0 Nov 01 22:16:19.719931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.719906 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" Nov 01 22:16:19.719931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.719919 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" event={"ID":"2db0c166-47aa-4aa8-be5d-f949cbddfbd7","Type":"ContainerDied","Data":"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304"} Nov 01 22:16:19.720307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.720011 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm" event={"ID":"2db0c166-47aa-4aa8-be5d-f949cbddfbd7","Type":"ContainerDied","Data":"f9c69636a4531e0c949a6cfd3bc8d679c9b5fef921c86350f99061adbf7c55a0"} Nov 01 22:16:19.720307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.720083 4395 scope.go:122] "RemoveContainer" containerID="edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304" Nov 01 22:16:19.735968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.735950 4395 scope.go:122] "RemoveContainer" containerID="edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304" Nov 01 22:16:19.736373 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:19.736339 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304\": container with ID starting with edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304 not found: ID does not exist" containerID="edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304" Nov 01 22:16:19.736452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.736380 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304"} err="failed to get container status \"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304\": rpc error: code = NotFound desc = could not find container \"edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304\": container with ID starting with edbd4d63db549ab027e97ada2047a8d66a3ddcaaff816c8b8ae9f021d1b0d304 not found: ID does not exist" Nov 01 22:16:19.750534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.750511 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:16:19.757984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:19.757960 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d8b5d-f9689554f-78pnm"] Nov 01 22:16:20.969698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:20.969659 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:16:21.168181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:21.168154 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" path="/var/lib/kubelet/pods/2db0c166-47aa-4aa8-be5d-f949cbddfbd7/volumes" Nov 01 22:16:22.390055 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:22.388252 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:16:25.375085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.374629 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:16:25.400438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.400409 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\") pod \"6d051c77-396c-435b-bdac-f5033e0dde02\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " Nov 01 22:16:25.400744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.400718 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume" pod "6d051c77-396c-435b-bdac-f5033e0dde02" (UID: "6d051c77-396c-435b-bdac-f5033e0dde02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:25.401005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.400986 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d051c77-396c-435b-bdac-f5033e0dde02-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:25.501918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.501875 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\" (UniqueName: \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\") pod \"6d051c77-396c-435b-bdac-f5033e0dde02\" (UID: \"6d051c77-396c-435b-bdac-f5033e0dde02\") " Nov 01 22:16:25.504418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.504387 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb" pod "6d051c77-396c-435b-bdac-f5033e0dde02" (UID: "6d051c77-396c-435b-bdac-f5033e0dde02"). InnerVolumeSpecName "kube-api-access-ql9nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:25.603758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.603726 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9nb\" (UniqueName: \"kubernetes.io/projected/6d051c77-396c-435b-bdac-f5033e0dde02-kube-api-access-ql9nb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:25.848763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.848674 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d051c77-396c-435b-bdac-f5033e0dde02" containerID="57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9" exitCode=0 Nov 01 22:16:25.848763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.848716 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" Nov 01 22:16:25.849001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.848752 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" event={"ID":"6d051c77-396c-435b-bdac-f5033e0dde02","Type":"ContainerDied","Data":"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9"} Nov 01 22:16:25.849001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.848819 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm" event={"ID":"6d051c77-396c-435b-bdac-f5033e0dde02","Type":"ContainerDied","Data":"0291744e0cbe1dfa026171cbadffb43cccbc78cf9e0666fd7c767f58b8adb134"} Nov 01 22:16:25.849001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.848849 4395 scope.go:122] "RemoveContainer" containerID="57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9" Nov 01 22:16:25.861856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.861837 4395 scope.go:122] "RemoveContainer" containerID="57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9" Nov 01 22:16:25.862262 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:25.862241 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9\": container with ID starting with 57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9 not found: ID does not exist" containerID="57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9" Nov 01 22:16:25.862343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.862272 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9"} err="failed to get container status \"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9\": rpc error: code = NotFound desc = could not find container \"57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9\": container with ID starting with 57a67e82dd6d9b6929240fe9e21487926896c1937fd9374bdcaf861b84e367e9 not found: ID does not exist" Nov 01 22:16:25.880859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.880833 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:16:25.885624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:25.885604 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-17957f-5686759fdb-fcrcm"] Nov 01 22:16:27.170079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:27.170049 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" path="/var/lib/kubelet/pods/6d051c77-396c-435b-bdac-f5033e0dde02/volumes" Nov 01 22:16:28.050055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:28.039415 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:16:28.050055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:28.039678 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" containerID="cri-o://d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75" gracePeriod=30 Nov 01 22:16:28.400098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:28.368341 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:16:28.400098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:28.368597 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" containerID="cri-o://d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc" gracePeriod=30 Nov 01 22:16:29.271666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:29.271634 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:16:29.271924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:29.271887 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" containerID="cri-o://8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15" gracePeriod=30 Nov 01 22:16:29.392930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:29.392875 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:16:30.596114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.595750 4395 patch_prober.go:29] interesting pod/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:16:30.596114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.595820 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:16:30.702987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.702959 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:16:30.730302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.730272 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\") pod \"d516c35b-22c9-44c9-b704-f79b10ff516d\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " Nov 01 22:16:30.730584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.730478 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\" (UniqueName: \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\") pod \"d516c35b-22c9-44c9-b704-f79b10ff516d\" (UID: \"d516c35b-22c9-44c9-b704-f79b10ff516d\") " Nov 01 22:16:30.731879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.731849 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume" pod "d516c35b-22c9-44c9-b704-f79b10ff516d" (UID: "d516c35b-22c9-44c9-b704-f79b10ff516d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:30.737878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.736805 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m" pod "d516c35b-22c9-44c9-b704-f79b10ff516d" (UID: "d516c35b-22c9-44c9-b704-f79b10ff516d"). InnerVolumeSpecName "kube-api-access-2tk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:30.843284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.837338 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tk9m\" (UniqueName: \"kubernetes.io/projected/d516c35b-22c9-44c9-b704-f79b10ff516d-kube-api-access-2tk9m\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:30.843284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:30.837378 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d516c35b-22c9-44c9-b704-f79b10ff516d-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:31.133103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.133009 4395 generic.go:356] "Generic (PLEG): container finished" podID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerID="4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276" exitCode=0 Nov 01 22:16:31.133103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.133069 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" Nov 01 22:16:31.133283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.133096 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" event={"ID":"d516c35b-22c9-44c9-b704-f79b10ff516d","Type":"ContainerDied","Data":"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276"} Nov 01 22:16:31.133283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.133153 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9" event={"ID":"d516c35b-22c9-44c9-b704-f79b10ff516d","Type":"ContainerDied","Data":"cf2cc552e7b89b3f829755d4a10e20726f77740e5e7bddc16311a839d071a7bf"} Nov 01 22:16:31.133283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.133175 4395 scope.go:122] "RemoveContainer" containerID="4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276" Nov 01 22:16:31.145333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.145312 4395 scope.go:122] "RemoveContainer" containerID="4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276" Nov 01 22:16:31.145610 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:31.145590 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276\": container with ID starting with 4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276 not found: ID does not exist" containerID="4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276" Nov 01 22:16:31.145682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.145620 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276"} err="failed to get container status \"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276\": rpc error: code = NotFound desc = could not find container \"4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276\": container with ID starting with 4124a1424eaa5e2d2da90fa2de2327ca679e2b8123883f22d7597eba539d9276 not found: ID does not exist" Nov 01 22:16:31.170865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.170839 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:16:31.171003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:31.170876 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d8b5d-64f95b8d99-n9vp9"] Nov 01 22:16:33.168749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.168717 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" path="/var/lib/kubelet/pods/d516c35b-22c9-44c9-b704-f79b10ff516d/volumes" Nov 01 22:16:33.578894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.578866 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:16:33.657592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.657560 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.657745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.657627 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.657745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.657727 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.657911 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.657970 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658009 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658090 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658145 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658257 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\") pod \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\" (UID: \"66c6c593-66d7-4edb-ae89-0f4e1fd13365\") " Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658615 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658712 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.658942 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.659290 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.659534 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.660132 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.660157 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.660172 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.660187 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.660233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.660201 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.661389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.661146 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:33.661562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.661538 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "kube-api-access-vgnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:33.662940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.662918 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.663330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.663306 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token" pod "66c6c593-66d7-4edb-ae89-0f4e1fd13365" (UID: "66c6c593-66d7-4edb-ae89-0f4e1fd13365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:33.761906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.761877 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.761906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.761907 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.762158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.761916 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c6c593-66d7-4edb-ae89-0f4e1fd13365-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.762158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.761926 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgnrs\" (UniqueName: \"kubernetes.io/projected/66c6c593-66d7-4edb-ae89-0f4e1fd13365-kube-api-access-vgnrs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.938463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.938433 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:16:33.965383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965351 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965398 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965461 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965562 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965725 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965764 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.965834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965813 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.966878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965879 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.966878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\") pod \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\" (UID: \"c4a95cf6-2b84-4c29-980d-a0e0cc64557f\") " Nov 01 22:16:33.966878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.965999 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.967160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.967135 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:33.967302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.967179 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.967414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.967342 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.967414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.967384 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:33.968350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.966235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:33.968746 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.968701 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:33.968992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.968972 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:33.969299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.969274 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "kube-api-access-bw4f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:33.969415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:33.969376 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy" pod "c4a95cf6-2b84-4c29-980d-a0e0cc64557f" (UID: "c4a95cf6-2b84-4c29-980d-a0e0cc64557f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068727 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068754 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068765 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068774 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068782 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.068795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068790 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.069113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068800 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.069113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.068810 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4f2\" (UniqueName: \"kubernetes.io/projected/c4a95cf6-2b84-4c29-980d-a0e0cc64557f-kube-api-access-bw4f2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.234941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.234905 4395 generic.go:356] "Generic (PLEG): container finished" podID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerID="d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc" exitCode=0 Nov 01 22:16:34.235561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.234965 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" Nov 01 22:16:34.235561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.234979 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" event={"ID":"c4a95cf6-2b84-4c29-980d-a0e0cc64557f","Type":"ContainerDied","Data":"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc"} Nov 01 22:16:34.235561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.235086 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw" event={"ID":"c4a95cf6-2b84-4c29-980d-a0e0cc64557f","Type":"ContainerDied","Data":"8bef07a1ab460bc853cdbc4c28ad75fef53e9f3a0d8cea5bb9c6a0d63950ae97"} Nov 01 22:16:34.235561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.235153 4395 scope.go:122] "RemoveContainer" containerID="d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc" Nov 01 22:16:34.241309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.241284 4395 generic.go:356] "Generic (PLEG): container finished" podID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerID="d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75" exitCode=0 Nov 01 22:16:34.241443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.241343 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" event={"ID":"66c6c593-66d7-4edb-ae89-0f4e1fd13365","Type":"ContainerDied","Data":"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75"} Nov 01 22:16:34.241443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.241344 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" Nov 01 22:16:34.241443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.241376 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5" event={"ID":"66c6c593-66d7-4edb-ae89-0f4e1fd13365","Type":"ContainerDied","Data":"37d993258c274a2b679bc5b710a6e6a83a16ebe56ae09be6da78cb444ef895c9"} Nov 01 22:16:34.252811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.252777 4395 scope.go:122] "RemoveContainer" containerID="d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc" Nov 01 22:16:34.253112 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:34.253090 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc\": container with ID starting with d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc not found: ID does not exist" containerID="d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc" Nov 01 22:16:34.253222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.253120 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc"} err="failed to get container status \"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc\": rpc error: code = NotFound desc = could not find container \"d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc\": container with ID starting with d3bfa8b2cb49f1af46c35cacb6d693f7f97c7ee10ba8b64f0050a40a742a26fc not found: ID does not exist" Nov 01 22:16:34.253222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.253137 4395 scope.go:122] "RemoveContainer" containerID="d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75" Nov 01 22:16:34.264355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.264281 4395 scope.go:122] "RemoveContainer" containerID="d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75" Nov 01 22:16:34.268088 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:34.267967 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75\": container with ID starting with d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75 not found: ID does not exist" containerID="d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75" Nov 01 22:16:34.268189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.268016 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75"} err="failed to get container status \"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75\": rpc error: code = NotFound desc = could not find container \"d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75\": container with ID starting with d8e5283e419df3eb6c47097776debb2899505344527b106ea28205a154242c75 not found: ID does not exist" Nov 01 22:16:34.268720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.268696 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:16:34.273738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.273719 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-fb2cw"] Nov 01 22:16:34.286445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.286422 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:16:34.291119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.291099 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-544w5"] Nov 01 22:16:34.585597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.585576 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:16:34.724567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724476 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724560 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724618 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724658 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724739 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724896 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.724965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724921 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.724965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.724927 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.725177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725108 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.725177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725082 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.725177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725146 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\") pod \"1c5d8060-ec8b-4828-95f2-6da59624e20b\" (UID: \"1c5d8060-ec8b-4828-95f2-6da59624e20b\") " Nov 01 22:16:34.725177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725151 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.725383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725293 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.725508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.725484 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:34.726264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.726242 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.726361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.726272 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.726361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.726289 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.726361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.726306 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.726361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.726321 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5d8060-ec8b-4828-95f2-6da59624e20b-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.727302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.727270 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:34.727406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.727331 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:34.727529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.727511 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "kube-api-access-gwtl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:34.727608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.727586 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token" pod "1c5d8060-ec8b-4828-95f2-6da59624e20b" (UID: "1c5d8060-ec8b-4828-95f2-6da59624e20b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:34.827621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.827587 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtl6\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-kube-api-access-gwtl6\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.827621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.827618 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.827804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.827628 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:34.827804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:34.827638 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5d8060-ec8b-4828-95f2-6da59624e20b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:35.173495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.173453 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" path="/var/lib/kubelet/pods/66c6c593-66d7-4edb-ae89-0f4e1fd13365/volumes" Nov 01 22:16:35.174250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.174229 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" path="/var/lib/kubelet/pods/c4a95cf6-2b84-4c29-980d-a0e0cc64557f/volumes" Nov 01 22:16:35.272275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.272239 4395 generic.go:356] "Generic (PLEG): container finished" podID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerID="8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15" exitCode=0 Nov 01 22:16:35.273019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.272830 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" Nov 01 22:16:35.277782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.272853 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" event={"ID":"1c5d8060-ec8b-4828-95f2-6da59624e20b","Type":"ContainerDied","Data":"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15"} Nov 01 22:16:35.278094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.278072 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s" event={"ID":"1c5d8060-ec8b-4828-95f2-6da59624e20b","Type":"ContainerDied","Data":"88235f6cbe1811e7098308e8c7aaf80811240d88cd7c0a7ced16175ceeb1c8fa"} Nov 01 22:16:35.279182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.279120 4395 scope.go:122] "RemoveContainer" containerID="8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15" Nov 01 22:16:35.383817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.383184 4395 scope.go:122] "RemoveContainer" containerID="8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15" Nov 01 22:16:35.386675 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:35.386537 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15\": container with ID starting with 8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15 not found: ID does not exist" containerID="8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15" Nov 01 22:16:35.386675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.386580 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15"} err="failed to get container status \"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15\": rpc error: code = NotFound desc = could not find container \"8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15\": container with ID starting with 8faed0783e1918b9b6fb2aac1b86d0c95e8b4e11055154d957d32b704e1cca15 not found: ID does not exist" Nov 01 22:16:35.386675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.386626 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:16:35.450101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:35.449702 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8483c2-openshift-default-86687957bf-hxs6s"] Nov 01 22:16:37.167265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:37.167235 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" path="/var/lib/kubelet/pods/1c5d8060-ec8b-4828-95f2-6da59624e20b/volumes" Nov 01 22:16:44.640918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:44.640875 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:16:44.641372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:44.641149 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerName="maas-api" containerID="cri-o://92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe" gracePeriod=30 Nov 01 22:16:45.342911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.342885 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:16:45.372047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.371996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\" (UniqueName: \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\") pod \"a4b69845-1b1d-442a-b32c-3071809ba49f\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " Nov 01 22:16:45.372214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.372172 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\") pod \"a4b69845-1b1d-442a-b32c-3071809ba49f\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " Nov 01 22:16:45.372285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.372225 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\") pod \"a4b69845-1b1d-442a-b32c-3071809ba49f\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " Nov 01 22:16:45.372344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.372294 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\") pod \"a4b69845-1b1d-442a-b32c-3071809ba49f\" (UID: \"a4b69845-1b1d-442a-b32c-3071809ba49f\") " Nov 01 22:16:45.372777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.372733 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca" pod "a4b69845-1b1d-442a-b32c-3071809ba49f" (UID: "a4b69845-1b1d-442a-b32c-3071809ba49f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:45.373420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.373393 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b69845-1b1d-442a-b32c-3071809ba49f-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:45.374913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.374854 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls" pod "a4b69845-1b1d-442a-b32c-3071809ba49f" (UID: "a4b69845-1b1d-442a-b32c-3071809ba49f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:45.376214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.376188 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs" pod "a4b69845-1b1d-442a-b32c-3071809ba49f" (UID: "a4b69845-1b1d-442a-b32c-3071809ba49f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:16:45.376346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.376228 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg" pod "a4b69845-1b1d-442a-b32c-3071809ba49f" (UID: "a4b69845-1b1d-442a-b32c-3071809ba49f"). InnerVolumeSpecName "kube-api-access-nhjvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:45.410886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.410851 4395 generic.go:356] "Generic (PLEG): container finished" podID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerID="92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe" exitCode=0 Nov 01 22:16:45.411073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.410902 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" Nov 01 22:16:45.411073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.410929 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" event={"ID":"a4b69845-1b1d-442a-b32c-3071809ba49f","Type":"ContainerDied","Data":"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe"} Nov 01 22:16:45.411073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.411013 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" event={"ID":"a4b69845-1b1d-442a-b32c-3071809ba49f","Type":"ContainerDied","Data":"5b3d0cdd17dac6471e637fa19c295d5daac21eb8e3958c58a17490a5fe9f14a8"} Nov 01 22:16:45.411073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.411059 4395 scope.go:122] "RemoveContainer" containerID="92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe" Nov 01 22:16:45.423356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.423339 4395 scope.go:122] "RemoveContainer" containerID="92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe" Nov 01 22:16:45.423702 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:45.423682 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe\": container with ID starting with 92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe not found: ID does not exist" containerID="92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe" Nov 01 22:16:45.423787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.423710 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe"} err="failed to get container status \"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe\": rpc error: code = NotFound desc = could not find container \"92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe\": container with ID starting with 92a11603c748a7339f2a43a4863695e278307552b6f6a354b6b15c900f6f7bfe not found: ID does not exist" Nov 01 22:16:45.441260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.441229 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:16:45.445740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.445718 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv"] Nov 01 22:16:45.483497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.483322 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjvg\" (UniqueName: \"kubernetes.io/projected/a4b69845-1b1d-442a-b32c-3071809ba49f-kube-api-access-nhjvg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:45.487234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.487204 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:45.488231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:45.487368 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b69845-1b1d-442a-b32c-3071809ba49f-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:47.169663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:47.169631 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" path="/var/lib/kubelet/pods/a4b69845-1b1d-442a-b32c-3071809ba49f/volumes" Nov 01 22:16:47.935078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:47.934995 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-830149-65bcbf6998-x28xv" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.139:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:16:52.390054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:52.389991 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:16:52.390738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:52.390255 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" containerID="cri-o://ed532ffc91cab403612fb2870e4298afabc3be2d2a218fff220a7860e23160ab" gracePeriod=30 Nov 01 22:16:52.432080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:52.429692 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:16:52.432080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:52.429985 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" containerID="cri-o://a13ebecf7d08cb166989007f59c8c6be79c90c39641386acf52cbe7d7ef8aab9" gracePeriod=30 Nov 01 22:16:52.510456 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:16:52.510406 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:16:57.675951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.675534 4395 generic.go:356] "Generic (PLEG): container finished" podID="07051168-24a0-49a9-a528-cc8098bf2692" containerID="a13ebecf7d08cb166989007f59c8c6be79c90c39641386acf52cbe7d7ef8aab9" exitCode=0 Nov 01 22:16:57.675951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.675800 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerDied","Data":"a13ebecf7d08cb166989007f59c8c6be79c90c39641386acf52cbe7d7ef8aab9"} Nov 01 22:16:57.675951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.675847 4395 scope.go:122] "RemoveContainer" containerID="f98bde87b5f0bf246ce431c7235a1fe5003a65cb1b215cf2346dc80abf9681aa" Nov 01 22:16:57.678288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.678264 4395 generic.go:356] "Generic (PLEG): container finished" podID="68db399d-6ade-4a07-92af-3cfafb61d690" containerID="ed532ffc91cab403612fb2870e4298afabc3be2d2a218fff220a7860e23160ab" exitCode=0 Nov 01 22:16:57.678417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.678319 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" event={"ID":"68db399d-6ade-4a07-92af-3cfafb61d690","Type":"ContainerDied","Data":"ed532ffc91cab403612fb2870e4298afabc3be2d2a218fff220a7860e23160ab"} Nov 01 22:16:57.864763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.864739 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:16:57.868071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:57.868047 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:16:58.026630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026589 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.026630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026635 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026700 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026757 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026803 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026863 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026874 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.026915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026903 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026936 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.026998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027020 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027061 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027135 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027235 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027330 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027386 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027403 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027429 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027453 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027577 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\") pod \"68db399d-6ade-4a07-92af-3cfafb61d690\" (UID: \"68db399d-6ade-4a07-92af-3cfafb61d690\") " Nov 01 22:16:58.027635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027619 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.028019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027623 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.028019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027682 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\") pod \"07051168-24a0-49a9-a528-cc8098bf2692\" (UID: \"07051168-24a0-49a9-a528-cc8098bf2692\") " Nov 01 22:16:58.028019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027773 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:58.028019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027795 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.028019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.027937 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.028318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028200 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.028462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028438 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:16:58.028937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028909 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028940 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028956 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028969 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028983 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07051168-24a0-49a9-a528-cc8098bf2692-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.028998 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.029013 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.029065 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.029081 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68db399d-6ade-4a07-92af-3cfafb61d690-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.029404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.029094 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.030582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.030556 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.030682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.030632 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.030948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.030917 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "kube-api-access-dhcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.031126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.030977 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.031209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.031185 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo" pod "07051168-24a0-49a9-a528-cc8098bf2692" (UID: "07051168-24a0-49a9-a528-cc8098bf2692"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:58.031433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.031414 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "kube-api-access-qjxt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:16:58.032021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.031996 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:16:58.032204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.032188 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy" pod "68db399d-6ade-4a07-92af-3cfafb61d690" (UID: "68db399d-6ade-4a07-92af-3cfafb61d690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130337 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07051168-24a0-49a9-a528-cc8098bf2692-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130360 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07051168-24a0-49a9-a528-cc8098bf2692-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130368 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68db399d-6ade-4a07-92af-3cfafb61d690-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130377 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68db399d-6ade-4a07-92af-3cfafb61d690-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130385 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130393 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjxt6\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-kube-api-access-qjxt6\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130402 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcmc\" (UniqueName: \"kubernetes.io/projected/07051168-24a0-49a9-a528-cc8098bf2692-kube-api-access-dhcmc\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.130688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.130410 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68db399d-6ade-4a07-92af-3cfafb61d690-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:16:58.687479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.687452 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" Nov 01 22:16:58.687479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.687468 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww" event={"ID":"68db399d-6ade-4a07-92af-3cfafb61d690","Type":"ContainerDied","Data":"9c1eff50d1d3f2bd1c7e94ec6ccc4d2b1693bb61c57a1dcafa490de47afebd3e"} Nov 01 22:16:58.688098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.687567 4395 scope.go:122] "RemoveContainer" containerID="ed532ffc91cab403612fb2870e4298afabc3be2d2a218fff220a7860e23160ab" Nov 01 22:16:58.689279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.689253 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" event={"ID":"07051168-24a0-49a9-a528-cc8098bf2692","Type":"ContainerDied","Data":"d0d710494ab0c71239f2de2934740ac36bad183e7ff3355ae9858f38702d63ec"} Nov 01 22:16:58.689279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.689271 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz" Nov 01 22:16:58.701020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.701004 4395 scope.go:122] "RemoveContainer" containerID="a13ebecf7d08cb166989007f59c8c6be79c90c39641386acf52cbe7d7ef8aab9" Nov 01 22:16:58.720510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.720488 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:16:58.724600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.724579 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-774ww"] Nov 01 22:16:58.737385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.737367 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:16:58.743153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:58.743131 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-2pqqz"] Nov 01 22:16:59.166552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:59.166523 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07051168-24a0-49a9-a528-cc8098bf2692" path="/var/lib/kubelet/pods/07051168-24a0-49a9-a528-cc8098bf2692/volumes" Nov 01 22:16:59.167135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:16:59.167119 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" path="/var/lib/kubelet/pods/68db399d-6ade-4a07-92af-3cfafb61d690/volumes" Nov 01 22:17:06.226919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:06.226882 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:17:11.228004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:11.227966 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" containerID="cri-o://de9caf17fb14941c66e2b7e9a2ad801b2b58631d49fed5e6c82fb2b4fb6b2629" gracePeriod=25 Nov 01 22:17:13.992045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:13.991991 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:17:13.992518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:13.992286 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" podUID="c5a699c8-3443-4993-a7d6-74d8deb98576" containerName="maas-api" containerID="cri-o://ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3" gracePeriod=30 Nov 01 22:17:14.616215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.616190 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:17:14.809342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.809303 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\") pod \"c5a699c8-3443-4993-a7d6-74d8deb98576\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " Nov 01 22:17:14.809642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.809355 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\" (UniqueName: \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\") pod \"c5a699c8-3443-4993-a7d6-74d8deb98576\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " Nov 01 22:17:14.809642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.809479 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\") pod \"c5a699c8-3443-4993-a7d6-74d8deb98576\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " Nov 01 22:17:14.809642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.809630 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\") pod \"c5a699c8-3443-4993-a7d6-74d8deb98576\" (UID: \"c5a699c8-3443-4993-a7d6-74d8deb98576\") " Nov 01 22:17:14.810292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.810153 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca" pod "c5a699c8-3443-4993-a7d6-74d8deb98576" (UID: "c5a699c8-3443-4993-a7d6-74d8deb98576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:14.810461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.810442 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a699c8-3443-4993-a7d6-74d8deb98576-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:14.811892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.811866 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs" pod "c5a699c8-3443-4993-a7d6-74d8deb98576" (UID: "c5a699c8-3443-4993-a7d6-74d8deb98576"). InnerVolumeSpecName "kube-api-access-hdhqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:14.812163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.812131 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls" pod "c5a699c8-3443-4993-a7d6-74d8deb98576" (UID: "c5a699c8-3443-4993-a7d6-74d8deb98576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:17:14.812163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.812142 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs" pod "c5a699c8-3443-4993-a7d6-74d8deb98576" (UID: "c5a699c8-3443-4993-a7d6-74d8deb98576"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:17:14.886073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.885138 4395 generic.go:356] "Generic (PLEG): container finished" podID="c5a699c8-3443-4993-a7d6-74d8deb98576" containerID="ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3" exitCode=0 Nov 01 22:17:14.886073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.885387 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" event={"ID":"c5a699c8-3443-4993-a7d6-74d8deb98576","Type":"ContainerDied","Data":"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3"} Nov 01 22:17:14.886073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.885443 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" event={"ID":"c5a699c8-3443-4993-a7d6-74d8deb98576","Type":"ContainerDied","Data":"2bb70cfdb744a48a63ac56ddba6e96e7f954caa7daf0c2548b3d868c16ae260e"} Nov 01 22:17:14.886073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.885471 4395 scope.go:122] "RemoveContainer" containerID="ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3" Nov 01 22:17:14.886073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.885625 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n" Nov 01 22:17:14.920060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.918936 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:14.920060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.918991 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a699c8-3443-4993-a7d6-74d8deb98576-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:14.920060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.919006 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhqs\" (UniqueName: \"kubernetes.io/projected/c5a699c8-3443-4993-a7d6-74d8deb98576-kube-api-access-hdhqs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:14.941082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.939219 4395 scope.go:122] "RemoveContainer" containerID="ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3" Nov 01 22:17:14.942324 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:14.942052 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3\": container with ID starting with ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3 not found: ID does not exist" containerID="ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3" Nov 01 22:17:14.942324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.942095 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3"} err="failed to get container status \"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3\": rpc error: code = NotFound desc = could not find container \"ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3\": container with ID starting with ea417e95bab9e6af8537f7428087c9fb2b32f2f350c21c1144175c0a55f2b4e3 not found: ID does not exist" Nov 01 22:17:14.948542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.948516 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:17:14.953432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:14.953410 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-487639-688874b489-png2n"] Nov 01 22:17:15.169549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:15.169463 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a699c8-3443-4993-a7d6-74d8deb98576" path="/var/lib/kubelet/pods/c5a699c8-3443-4993-a7d6-74d8deb98576/volumes" Nov 01 22:17:15.194905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:15.194862 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:17:19.671233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671204 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:17:19.671942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671928 4395 cpu_manager.go:414] "containerMap: removing container" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671943 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671956 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671961 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671970 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" containerName="maas-api" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671975 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" containerName="maas-api" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671983 4395 cpu_manager.go:414] "containerMap: removing container" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.671987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.671987 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672000 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672004 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672012 4395 cpu_manager.go:414] "containerMap: removing container" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672017 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672050 4395 cpu_manager.go:414] "containerMap: removing container" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672057 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672070 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672075 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672089 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672094 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672104 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672109 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672127 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672131 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672145 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c5a699c8-3443-4993-a7d6-74d8deb98576" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672149 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a699c8-3443-4993-a7d6-74d8deb98576" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672255 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a699c8-3443-4993-a7d6-74d8deb98576" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672264 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672270 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d051c77-396c-435b-bdac-f5033e0dde02" containerName="payload-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672277 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db0c166-47aa-4aa8-be5d-f949cbddfbd7" containerName="maas-api" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672282 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="07051168-24a0-49a9-a528-cc8098bf2692" containerName="istio-proxy" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672288 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d516c35b-22c9-44c9-b704-f79b10ff516d" containerName="payload-pre-processing" Nov 01 22:17:19.672282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672294 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c6c593-66d7-4edb-ae89-0f4e1fd13365" containerName="istio-proxy" Nov 01 22:17:19.672938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672301 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="68db399d-6ade-4a07-92af-3cfafb61d690" containerName="istio-proxy" Nov 01 22:17:19.672938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672306 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b69845-1b1d-442a-b32c-3071809ba49f" containerName="maas-api" Nov 01 22:17:19.672938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672314 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5d8060-ec8b-4828-95f2-6da59624e20b" containerName="istio-proxy" Nov 01 22:17:19.672938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672321 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c54621-69e3-4946-9ef9-ccf05fff0199" containerName="payload-pre-processing" Nov 01 22:17:19.672938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.672328 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a95cf6-2b84-4c29-980d-a0e0cc64557f" containerName="istio-proxy" Nov 01 22:17:19.678084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.678066 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:19.684877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.684854 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-51e3b7\"" Nov 01 22:17:19.686640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.686617 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:17:19.824052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.816421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:19.852414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.852376 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddsz\" (UniqueName: \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:19.966301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.966209 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 22:17:19.966848 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.966810 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" containerID="cri-o://e2e2e4202294e1642b5926dec2daaa4ea84759579807a628c70ee6afedbd4fd9" gracePeriod=30 Nov 01 22:17:19.981095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.980224 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:19.981095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:19.980426 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddsz\" (UniqueName: \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:20.000588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.000543 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:20.008702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.003846 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 22:17:20.008702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.004118 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" containerID="cri-o://a4f62600334ccaa65fd33e535e7fdd4090812ca0b4ae28500b9bedc2134ff876" gracePeriod=30 Nov 01 22:17:20.019203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.017396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddsz\" (UniqueName: \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\") pod \"payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:20.294351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.294311 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 22:17:20.294605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.294578 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" containerID="cri-o://e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94" gracePeriod=30 Nov 01 22:17:20.297263 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.297239 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:20.651074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:20.651045 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:17:20.651513 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:17:20.651488 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a0e765_4dad_4a0a_ae48_9b629641ed23.slice/crio-6244b126f868acab76ccdf09792b2a9939cc33d9eb2dabfd4fa20ff761846b41 WatchSource:0}: Error finding container 6244b126f868acab76ccdf09792b2a9939cc33d9eb2dabfd4fa20ff761846b41: Status 404 returned error can't find the container with id 6244b126f868acab76ccdf09792b2a9939cc33d9eb2dabfd4fa20ff761846b41 Nov 01 22:17:21.098468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:21.098426 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" event={"ID":"09a0e765-4dad-4a0a-ae48-9b629641ed23","Type":"ContainerStarted","Data":"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b"} Nov 01 22:17:21.098869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:21.098497 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" event={"ID":"09a0e765-4dad-4a0a-ae48-9b629641ed23","Type":"ContainerStarted","Data":"6244b126f868acab76ccdf09792b2a9939cc33d9eb2dabfd4fa20ff761846b41"} Nov 01 22:17:21.098869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:21.098534 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:21.122497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:21.122408 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" podStartSLOduration=2.122385398 podStartE2EDuration="2.122385398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:21.116750123 +0000 UTC m=+1988.090652663" watchObservedRunningTime="2026-11-01 22:17:21.122385398 +0000 UTC m=+1988.096287937" Nov 01 22:17:22.354734 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:22.354697 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:17:24.668511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:24.668475 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:24.668902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:24.668531 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:25.269479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.269287 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:17:25.269686 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.269659 4395 generic.go:356] "Generic (PLEG): container finished" podID="ec7185eb-26b9-4542-9552-17cd22207048" containerID="a4f62600334ccaa65fd33e535e7fdd4090812ca0b4ae28500b9bedc2134ff876" exitCode=0 Nov 01 22:17:25.269876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.269696 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerDied","Data":"a4f62600334ccaa65fd33e535e7fdd4090812ca0b4ae28500b9bedc2134ff876"} Nov 01 22:17:25.269876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.269754 4395 scope.go:122] "RemoveContainer" containerID="c51d04faf14256c305a28f0f1ee8fdee90732696b88977e6a21129bd95d7bd51" Nov 01 22:17:25.272099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.272008 4395 generic.go:356] "Generic (PLEG): container finished" podID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerID="e2e2e4202294e1642b5926dec2daaa4ea84759579807a628c70ee6afedbd4fd9" exitCode=0 Nov 01 22:17:25.272099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.272066 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" event={"ID":"e11149a0-fad0-4266-9944-0f88b9b5098e","Type":"ContainerDied","Data":"e2e2e4202294e1642b5926dec2daaa4ea84759579807a628c70ee6afedbd4fd9"} Nov 01 22:17:25.351871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.351836 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:17:25.352000 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.351895 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 22:17:25.621256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.621231 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.667702 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.667817 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.667956 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668022 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668063 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668173 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668231 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668265 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.669374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.668352 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\") pod \"247cb386-b126-4583-8b2f-cb0409a89d9b\" (UID: \"247cb386-b126-4583-8b2f-cb0409a89d9b\") " Nov 01 22:17:25.673897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.671083 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.673897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.672517 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.673897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.672734 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.673897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.672788 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.673897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.672827 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:25.711506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.711433 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:25.711971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.711925 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.712304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.712241 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "kube-api-access-g7m8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:25.716288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.716255 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo" pod "247cb386-b126-4583-8b2f-cb0409a89d9b" (UID: "247cb386-b126-4583-8b2f-cb0409a89d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:25.754067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.753604 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 22:17:25.772630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.771142 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.772630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.771210 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.772630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.771261 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.772630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.771295 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.772630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.771435 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.808527 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.809334 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.809422 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.809533 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.809561 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\") pod \"e11149a0-fad0-4266-9944-0f88b9b5098e\" (UID: \"e11149a0-fad0-4266-9944-0f88b9b5098e\") " Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810425 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247cb386-b126-4583-8b2f-cb0409a89d9b-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810447 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810461 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810474 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810488 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810501 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810514 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810528 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m8f\" (UniqueName: \"kubernetes.io/projected/247cb386-b126-4583-8b2f-cb0409a89d9b-kube-api-access-g7m8f\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810540 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.810553 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247cb386-b126-4583-8b2f-cb0409a89d9b-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.811470 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.812431 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.812707 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.813094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.812959 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.833502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.833425 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:25.833662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.833610 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:25.834451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.834410 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:25.835436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.835402 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85" pod "e11149a0-fad0-4266-9944-0f88b9b5098e" (UID: "e11149a0-fad0-4266-9944-0f88b9b5098e"). InnerVolumeSpecName "kube-api-access-rfk85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916178 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916215 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e11149a0-fad0-4266-9944-0f88b9b5098e-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916231 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916247 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916261 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916276 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e11149a0-fad0-4266-9944-0f88b9b5098e-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916293 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfk85\" (UniqueName: \"kubernetes.io/projected/e11149a0-fad0-4266-9944-0f88b9b5098e-kube-api-access-rfk85\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.916351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.916307 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e11149a0-fad0-4266-9944-0f88b9b5098e-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:25.930652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:25.930624 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.017414 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.017846 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.018069 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.018533 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.019104 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.019461 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.019594 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.019965 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.020349 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.020635 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.020714 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.020946 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.021145 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\") pod \"ec7185eb-26b9-4542-9552-17cd22207048\" (UID: \"ec7185eb-26b9-4542-9552-17cd22207048\") " Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.021384 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022335 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022361 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec7185eb-26b9-4542-9552-17cd22207048-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022381 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022397 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022414 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.022943 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.024846 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.025992 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "kube-api-access-5wmfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:26.027417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.026572 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token" pod "ec7185eb-26b9-4542-9552-17cd22207048" (UID: "ec7185eb-26b9-4542-9552-17cd22207048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:26.123664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.123628 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.123664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.123659 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec7185eb-26b9-4542-9552-17cd22207048-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.123664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.123672 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmfg\" (UniqueName: \"kubernetes.io/projected/ec7185eb-26b9-4542-9552-17cd22207048-kube-api-access-5wmfg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.123908 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.123681 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec7185eb-26b9-4542-9552-17cd22207048-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:26.318874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.318773 4395 generic.go:356] "Generic (PLEG): container finished" podID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerID="e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94" exitCode=0 Nov 01 22:17:26.319091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.318946 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" event={"ID":"247cb386-b126-4583-8b2f-cb0409a89d9b","Type":"ContainerDied","Data":"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94"} Nov 01 22:17:26.319091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.318984 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" event={"ID":"247cb386-b126-4583-8b2f-cb0409a89d9b","Type":"ContainerDied","Data":"8ec4def3004c50a13a73e1d156511a69898d3cdfc87ef37a3618c8c10e9dd049"} Nov 01 22:17:26.319091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.319007 4395 scope.go:122] "RemoveContainer" containerID="e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94" Nov 01 22:17:26.319614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.319403 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v" Nov 01 22:17:26.322274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.322254 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" Nov 01 22:17:26.322385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.322314 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9" event={"ID":"e11149a0-fad0-4266-9944-0f88b9b5098e","Type":"ContainerDied","Data":"c3106d99aec5fa3e4e9e2c312ac778542bf985fcc5ca7c360f76474525d063a9"} Nov 01 22:17:26.325318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.325284 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" event={"ID":"ec7185eb-26b9-4542-9552-17cd22207048","Type":"ContainerDied","Data":"fa8756064fa286d77c0628040d2bfe1860fa41252295a5df267ff9b5b1f93aaf"} Nov 01 22:17:26.325412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.325402 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z" Nov 01 22:17:26.335464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.335445 4395 scope.go:122] "RemoveContainer" containerID="e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94" Nov 01 22:17:26.335722 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:26.335701 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94\": container with ID starting with e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94 not found: ID does not exist" containerID="e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94" Nov 01 22:17:26.335786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.335729 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94"} err="failed to get container status \"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94\": rpc error: code = NotFound desc = could not find container \"e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94\": container with ID starting with e68630400e873a393f03397492a18c15139a8a4324d3dbed80cdd70525f05d94 not found: ID does not exist" Nov 01 22:17:26.335786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.335747 4395 scope.go:122] "RemoveContainer" containerID="e2e2e4202294e1642b5926dec2daaa4ea84759579807a628c70ee6afedbd4fd9" Nov 01 22:17:26.346335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.346316 4395 scope.go:122] "RemoveContainer" containerID="a4f62600334ccaa65fd33e535e7fdd4090812ca0b4ae28500b9bedc2134ff876" Nov 01 22:17:26.355167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.355048 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 22:17:26.360757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.360732 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5j56v"] Nov 01 22:17:26.374069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.374021 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 22:17:26.389221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.387384 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-9px6z"] Nov 01 22:17:26.406133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.406099 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 22:17:26.410285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:26.410258 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-qpqg9"] Nov 01 22:17:27.168089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:27.168051 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" path="/var/lib/kubelet/pods/247cb386-b126-4583-8b2f-cb0409a89d9b/volumes" Nov 01 22:17:27.168653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:27.168632 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" path="/var/lib/kubelet/pods/e11149a0-fad0-4266-9944-0f88b9b5098e/volumes" Nov 01 22:17:27.169126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:27.169112 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7185eb-26b9-4542-9552-17cd22207048" path="/var/lib/kubelet/pods/ec7185eb-26b9-4542-9552-17cd22207048/volumes" Nov 01 22:17:31.462699 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.462670 4395 generic.go:356] "Generic (PLEG): container finished" podID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerID="de9caf17fb14941c66e2b7e9a2ad801b2b58631d49fed5e6c82fb2b4fb6b2629" exitCode=0 Nov 01 22:17:31.463300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.462839 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" event={"ID":"defccf48-c732-4e39-9c1b-8d2f467609c3","Type":"ContainerDied","Data":"de9caf17fb14941c66e2b7e9a2ad801b2b58631d49fed5e6c82fb2b4fb6b2629"} Nov 01 22:17:31.603766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.603738 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:17:31.649832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.649795 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\" (UniqueName: \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\") pod \"defccf48-c732-4e39-9c1b-8d2f467609c3\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " Nov 01 22:17:31.650051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.650000 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\") pod \"defccf48-c732-4e39-9c1b-8d2f467609c3\" (UID: \"defccf48-c732-4e39-9c1b-8d2f467609c3\") " Nov 01 22:17:31.650568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.650539 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume" pod "defccf48-c732-4e39-9c1b-8d2f467609c3" (UID: "defccf48-c732-4e39-9c1b-8d2f467609c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:31.650832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.650810 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/defccf48-c732-4e39-9c1b-8d2f467609c3-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:31.652822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.652797 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72" pod "defccf48-c732-4e39-9c1b-8d2f467609c3" (UID: "defccf48-c732-4e39-9c1b-8d2f467609c3"). InnerVolumeSpecName "kube-api-access-gmk72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:31.752380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:31.752341 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmk72\" (UniqueName: \"kubernetes.io/projected/defccf48-c732-4e39-9c1b-8d2f467609c3-kube-api-access-gmk72\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:32.109877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.109809 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:17:32.475645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.475563 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" Nov 01 22:17:32.475645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.475596 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv" event={"ID":"defccf48-c732-4e39-9c1b-8d2f467609c3","Type":"ContainerDied","Data":"467c336c6e50a414ac82bcd72b6fa356faac87fbd4c8f4b6022b186a9271ad08"} Nov 01 22:17:32.476667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.475662 4395 scope.go:122] "RemoveContainer" containerID="de9caf17fb14941c66e2b7e9a2ad801b2b58631d49fed5e6c82fb2b4fb6b2629" Nov 01 22:17:32.507784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.507755 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:17:32.514201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:32.514171 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-487639-6bc59bb99f-nxltv"] Nov 01 22:17:33.169286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.169249 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" path="/var/lib/kubelet/pods/defccf48-c732-4e39-9c1b-8d2f467609c3/volumes" Nov 01 22:17:33.501148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.501112 4395 generic.go:356] "Generic (PLEG): container finished" podID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerID="6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad" exitCode=0 Nov 01 22:17:33.501670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.501201 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerDied","Data":"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad"} Nov 01 22:17:33.501847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.501824 4395 scope.go:122] "RemoveContainer" containerID="6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad" Nov 01 22:17:33.504877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.504591 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:33.504877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.504701 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:33.504877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.504783 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:33.543564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:33.542814 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:17:34.512543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:34.512507 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerStarted","Data":"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284"} Nov 01 22:17:35.258579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:35.258499 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:17:35.260143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:35.260105 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:35.260286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:35.260179 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:36.256099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:36.256063 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:36.256593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:36.256133 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:37.255655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.255619 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:37.255842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.255678 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.597247 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609218 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609266 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609284 4395 cpu_manager.go:414] "containerMap: removing container" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609291 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609380 4395 cpu_manager.go:414] "containerMap: removing container" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609389 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609410 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609417 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609617 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609627 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="247cb386-b126-4583-8b2f-cb0409a89d9b" containerName="istio-proxy" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609647 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="defccf48-c732-4e39-9c1b-8d2f467609c3" containerName="payload-pre-processing" Nov 01 22:17:37.613105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.609656 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11149a0-fad0-4266-9944-0f88b9b5098e" containerName="istio-proxy" Nov 01 22:17:37.617767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.617734 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.624895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.624086 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:17:37.632943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.632916 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-51e3b7-serving-cert\"" Nov 01 22:17:37.717068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.717017 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.717234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.717110 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zpp\" (UniqueName: \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.717234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.717152 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.717391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.717372 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.834051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.833145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.834051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.833350 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.834051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.833382 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zpp\" (UniqueName: \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.834051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.833418 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.834561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.834358 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.848090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.842844 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.853852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.852738 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.855867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.855845 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zpp\" (UniqueName: \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\") pod \"maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:37.940978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:37.940902 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:38.261048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:38.258820 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:38.261048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:38.258892 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:38.352060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:38.343936 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:17:38.543235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:38.543143 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" containerID="cri-o://4cff0fb6c98cc46f4f1ecfa39ff5bb696d580083432f295ac0975202e551fe03" gracePeriod=25 Nov 01 22:17:38.544273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:38.544235 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" event={"ID":"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32","Type":"ContainerStarted","Data":"80f79f42675118a59faed5c4143a460575e876056b1ce97ab5c96ec197294a8f"} Nov 01 22:17:39.098053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.093433 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:17:39.256046 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.256004 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:39.256187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.256076 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:39.629699 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.618868 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" event={"ID":"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32","Type":"ContainerStarted","Data":"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50"} Nov 01 22:17:39.629699 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.620087 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:39.706053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:39.705766 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" podStartSLOduration=2.19696359 podStartE2EDuration="2.705750748s" totalImagesPullingTime="508.787158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:39.703522955 +0000 UTC m=+2006.677425496" watchObservedRunningTime="2026-11-01 22:17:39.705750748 +0000 UTC m=+2006.679653286" Nov 01 22:17:40.256060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:40.256010 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 22:17:40.256491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:40.256081 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 22:17:41.261272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:41.261240 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:17:41.262460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:41.262439 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:17:44.139563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:44.139520 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:17:45.021077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.016995 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:17:45.021077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.018312 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:45.021077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.018333 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:45.021077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.018536 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7185eb-26b9-4542-9552-17cd22207048" containerName="istio-proxy" Nov 01 22:17:45.023451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.023427 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.044538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.042496 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:17:45.113072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.113041 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1783e3-openshift-default-dockercfg-kwmww\"" Nov 01 22:17:45.136865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.136829 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.136892 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.136942 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137099 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137141 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137228 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfml\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137279 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137350 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.137470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.137444 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239354 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239414 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfml\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239451 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239536 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239592 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239801 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.239882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239854 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.240347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239901 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.240347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.239986 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.240579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.240469 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.240737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.240717 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.247752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.245572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.247752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.245862 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.247752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.246473 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.247752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.246711 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.304790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.304406 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.305082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.304716 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.310741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.310711 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:17:45.312411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.312379 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfml\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.324358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.324330 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:17:45.324545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.324518 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.348758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.348729 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:45.362133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.362113 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e3890e-openshift-default-dockercfg-j98nz\"" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444144 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444286 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444334 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444380 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxpt\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444505 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444589 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.445500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.444684 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.545821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545787 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545833 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545863 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545888 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxpt\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545940 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.545969 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.546058 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.546175 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.546391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.546207 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.547281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.546427 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.547281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.546469 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.547549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.547524 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.547662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.547550 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.548064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.548004 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.548711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.548690 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.548926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.548911 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.558377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.558322 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.558467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.558447 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxpt\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.644951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.644925 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:17:45.645346 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:17:45.645325 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18e56e6_4a7b_49cb_a8e5_6e340e32e8d1.slice/crio-2fa18c74fc0bffc80bd6f8befe0fe2af55fdfc5f92474922d7deeb61920ecdf2 WatchSource:0}: Error finding container 2fa18c74fc0bffc80bd6f8befe0fe2af55fdfc5f92474922d7deeb61920ecdf2: Status 404 returned error can't find the container with id 2fa18c74fc0bffc80bd6f8befe0fe2af55fdfc5f92474922d7deeb61920ecdf2 Nov 01 22:17:45.648383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.648350 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:45.648472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.648424 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:45.648472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.648453 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:45.655282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.655260 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:45.668118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.668082 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" event={"ID":"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1","Type":"ContainerStarted","Data":"2fa18c74fc0bffc80bd6f8befe0fe2af55fdfc5f92474922d7deeb61920ecdf2"} Nov 01 22:17:45.961575 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:17:45.961550 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fc2ab6_2f90_42e8_8ac8_2701917ddd13.slice/crio-4d4407dec445a68f0badc2cc904c82d880384ae3fdbc5685b7415e5eabcc36f1 WatchSource:0}: Error finding container 4d4407dec445a68f0badc2cc904c82d880384ae3fdbc5685b7415e5eabcc36f1: Status 404 returned error can't find the container with id 4d4407dec445a68f0badc2cc904c82d880384ae3fdbc5685b7415e5eabcc36f1 Nov 01 22:17:45.963070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.963041 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:17:45.964703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.964672 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:45.964764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.964737 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:45.964797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:45.964765 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:17:46.641021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.640992 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:17:46.718291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.718148 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" event={"ID":"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13","Type":"ContainerStarted","Data":"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d"} Nov 01 22:17:46.718291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.718267 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" event={"ID":"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13","Type":"ContainerStarted","Data":"4d4407dec445a68f0badc2cc904c82d880384ae3fdbc5685b7415e5eabcc36f1"} Nov 01 22:17:46.722913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.722878 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" event={"ID":"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1","Type":"ContainerStarted","Data":"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c"} Nov 01 22:17:46.746629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.746581 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podStartSLOduration=1.746567464 podStartE2EDuration="1.746567464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.744899843 +0000 UTC m=+2013.718802380" watchObservedRunningTime="2026-11-01 22:17:46.746567464 +0000 UTC m=+2013.720470062" Nov 01 22:17:46.771189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:46.771128 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podStartSLOduration=2.771114745 podStartE2EDuration="2.771114745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.768873026 +0000 UTC m=+2013.742775566" watchObservedRunningTime="2026-11-01 22:17:46.771114745 +0000 UTC m=+2013.745017284" Nov 01 22:17:47.349185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.349159 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:47.350680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.350664 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:47.350837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.350820 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:47.659112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.658344 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:47.662249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.662221 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:47.662356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.662273 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:47.839051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.838913 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:17:47.839246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.839210 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" containerID="cri-o://4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115" gracePeriod=30 Nov 01 22:17:47.846640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.846610 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:17:47.846860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.846832 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" containerID="cri-o://7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a" gracePeriod=30 Nov 01 22:17:47.852059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.852017 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:17:47.852209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:47.852174 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" containerID="cri-o://976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6" gracePeriod=30 Nov 01 22:17:48.349774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:48.349734 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:48.349986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:48.349785 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:48.656158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:48.656066 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:48.656158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:48.656115 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:49.351053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:49.350141 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:49.351053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:49.350198 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:49.656004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:49.655891 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:49.656004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:49.655948 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:50.349218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.349185 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:50.349444 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.349241 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:50.656473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.656401 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:50.656919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.656893 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:50.872526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.870771 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:50.872526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:50.870827 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:51.356268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.354212 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:51.356268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.354276 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:51.356268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.355951 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:51.356268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.355991 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:51.435641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.435581 4395 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:17:51.435873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.435643 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:17:51.661152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.661058 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:51.661152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:51.661120 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:52.341686 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:52.341593 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:17:52.349610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:52.349578 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 01 22:17:52.349743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:52.349640 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 01 22:17:52.656216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:52.656123 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:17:52.656216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:52.656189 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:17:53.354947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.354920 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:53.355419 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.355394 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:53.356287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.356221 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:17:53.370170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.370149 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:17:53.375366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.375337 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:17:53.419481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419446 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419521 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419585 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419615 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419640 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419678 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.419714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419707 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419787 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419827 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419891 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419960 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.419991 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420022 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420179 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420275 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420341 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420376 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420427 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\") pod \"7edd1cae-0185-4d2c-88aa-d823651c7a45\" (UID: \"7edd1cae-0185-4d2c-88aa-d823651c7a45\") " Nov 01 22:17:53.420475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420468 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\") pod \"1adb9d11-b88e-487d-a352-1a0bd06e100d\" (UID: \"1adb9d11-b88e-487d-a352-1a0bd06e100d\") " Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.420724 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.421083 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.421384 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.421605 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.421789 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.421915 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422151 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422184 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422588 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422613 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422628 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7edd1cae-0185-4d2c-88aa-d823651c7a45-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422642 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422656 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422672 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422686 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adb9d11-b88e-487d-a352-1a0bd06e100d-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422700 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.422715 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.424136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.423694 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.425466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.425394 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "kube-api-access-t9mvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.433173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.432684 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.433569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.433367 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:53.433569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.433525 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "kube-api-access-vs55w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.438523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.438493 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token" pod "7edd1cae-0185-4d2c-88aa-d823651c7a45" (UID: "7edd1cae-0185-4d2c-88aa-d823651c7a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.445242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.445217 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.445919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.445884 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.447190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.447169 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo" pod "1adb9d11-b88e-487d-a352-1a0bd06e100d" (UID: "1adb9d11-b88e-487d-a352-1a0bd06e100d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525321 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525346 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs55w\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-kube-api-access-vs55w\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525357 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525366 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9mvv\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-kube-api-access-t9mvv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525374 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adb9d11-b88e-487d-a352-1a0bd06e100d-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525382 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adb9d11-b88e-487d-a352-1a0bd06e100d-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525391 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525399 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.528008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.525408 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7edd1cae-0185-4d2c-88aa-d823651c7a45-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.536546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.536523 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:17:53.626987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.626938 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627054 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627120 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627191 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627268 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627265 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.627426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627360 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627437 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627493 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.627718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.627616 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\") pod \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\" (UID: \"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62\") " Nov 01 22:17:53.628752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.628235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.628752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.628406 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.628752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.628430 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.628752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.628713 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:53.629187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.629150 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.629557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.629536 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.630200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.630165 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "kube-api-access-wd9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.635338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.635302 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:17:53.635515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.635489 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:17:53.636657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.636635 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token" pod "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" (UID: "ddbf8bf1-b905-4706-a6be-f23d5eb3bc62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:53.660519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.660494 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:53.661007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.660985 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:53.661644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.661622 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:17:53.729750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729715 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729747 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729761 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729773 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729786 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729797 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.729965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.729809 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9q6\" (UniqueName: \"kubernetes.io/projected/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62-kube-api-access-wd9q6\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:53.796646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.796561 4395 generic.go:356] "Generic (PLEG): container finished" podID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerID="4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115" exitCode=0 Nov 01 22:17:53.796646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.796604 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" Nov 01 22:17:53.796871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.796643 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" event={"ID":"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62","Type":"ContainerDied","Data":"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115"} Nov 01 22:17:53.796871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.796700 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb" event={"ID":"ddbf8bf1-b905-4706-a6be-f23d5eb3bc62","Type":"ContainerDied","Data":"4333b6ead0a0c04787c953ead9963cf1354e573304506a54c11133f628bbd1bb"} Nov 01 22:17:53.796871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.796732 4395 scope.go:122] "RemoveContainer" containerID="4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115" Nov 01 22:17:53.798563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.798536 4395 generic.go:356] "Generic (PLEG): container finished" podID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerID="976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6" exitCode=0 Nov 01 22:17:53.798695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.798610 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" event={"ID":"1adb9d11-b88e-487d-a352-1a0bd06e100d","Type":"ContainerDied","Data":"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6"} Nov 01 22:17:53.798695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.798663 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" event={"ID":"1adb9d11-b88e-487d-a352-1a0bd06e100d","Type":"ContainerDied","Data":"0647cd0341937b04b14700f5373376ad284f33ed5b909c19ad87c13ee62289de"} Nov 01 22:17:53.798695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.798671 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww" Nov 01 22:17:53.800489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.800466 4395 generic.go:356] "Generic (PLEG): container finished" podID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerID="7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a" exitCode=0 Nov 01 22:17:53.800624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.800536 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" Nov 01 22:17:53.800624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.800597 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" event={"ID":"7edd1cae-0185-4d2c-88aa-d823651c7a45","Type":"ContainerDied","Data":"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a"} Nov 01 22:17:53.800741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.800637 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl" event={"ID":"7edd1cae-0185-4d2c-88aa-d823651c7a45","Type":"ContainerDied","Data":"d5989aa14fbc7f615cb97a794d63cb5b9f63ace075649ebc029fd9cb58304bbe"} Nov 01 22:17:53.811363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.811338 4395 scope.go:122] "RemoveContainer" containerID="4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115" Nov 01 22:17:53.811681 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:53.811660 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115\": container with ID starting with 4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115 not found: ID does not exist" containerID="4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115" Nov 01 22:17:53.811775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.811688 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115"} err="failed to get container status \"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115\": rpc error: code = NotFound desc = could not find container \"4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115\": container with ID starting with 4fa4204d742454e9e0f234d96fb6753a50249ba4176514f1cafd0e3dc9ffb115 not found: ID does not exist" Nov 01 22:17:53.811775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.811707 4395 scope.go:122] "RemoveContainer" containerID="976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6" Nov 01 22:17:53.825826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.825805 4395 scope.go:122] "RemoveContainer" containerID="976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6" Nov 01 22:17:53.826172 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:53.826155 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6\": container with ID starting with 976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6 not found: ID does not exist" containerID="976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6" Nov 01 22:17:53.826251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.826180 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6"} err="failed to get container status \"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6\": rpc error: code = NotFound desc = could not find container \"976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6\": container with ID starting with 976489ea7de65b466ddf43448c22a4b1d372d359760fe07d3179d56e11b137f6 not found: ID does not exist" Nov 01 22:17:53.826251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.826197 4395 scope.go:122] "RemoveContainer" containerID="7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a" Nov 01 22:17:53.838959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.838932 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:17:53.843216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.843193 4395 scope.go:122] "RemoveContainer" containerID="7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a" Nov 01 22:17:53.843560 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:17:53.843534 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a\": container with ID starting with 7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a not found: ID does not exist" containerID="7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a" Nov 01 22:17:53.843641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.843570 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a"} err="failed to get container status \"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a\": rpc error: code = NotFound desc = could not find container \"7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a\": container with ID starting with 7e20f37827fe82ae11cd7ff897cf71d9a5065358fc1f3f843ff87bc329c0e77a not found: ID does not exist" Nov 01 22:17:53.847291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.847266 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-fzklb"] Nov 01 22:17:53.865766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.865666 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:17:53.870397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.870375 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-bmhzl"] Nov 01 22:17:53.877152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.877130 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:17:53.882007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:53.881985 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-wmpww"] Nov 01 22:17:54.140015 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:54.139928 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:17:55.168415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:55.168382 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" path="/var/lib/kubelet/pods/1adb9d11-b88e-487d-a352-1a0bd06e100d/volumes" Nov 01 22:17:55.168866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:55.168852 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" path="/var/lib/kubelet/pods/7edd1cae-0185-4d2c-88aa-d823651c7a45/volumes" Nov 01 22:17:55.169368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:55.169351 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" path="/var/lib/kubelet/pods/ddbf8bf1-b905-4706-a6be-f23d5eb3bc62/volumes" Nov 01 22:17:58.552476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.552447 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:17:58.553267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553246 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" Nov 01 22:17:58.553267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553264 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" Nov 01 22:17:58.553437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553292 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" Nov 01 22:17:58.553437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553298 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" Nov 01 22:17:58.553437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553309 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" Nov 01 22:17:58.553437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553317 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" Nov 01 22:17:58.553577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553452 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adb9d11-b88e-487d-a352-1a0bd06e100d" containerName="istio-proxy" Nov 01 22:17:58.553577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553467 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbf8bf1-b905-4706-a6be-f23d5eb3bc62" containerName="istio-proxy" Nov 01 22:17:58.553577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.553474 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edd1cae-0185-4d2c-88aa-d823651c7a45" containerName="istio-proxy" Nov 01 22:17:58.560370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.560351 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.581412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.581379 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-521734\"" Nov 01 22:17:58.581535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.581513 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-521734-dockercfg-2k92x\"" Nov 01 22:17:58.583563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.583538 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:17:58.800654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.751069 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.800654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.751535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pqn\" (UniqueName: \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.800654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.799579 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:17:58.812052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.808253 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.852648 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.861817 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgpk\" (UniqueName: \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.861862 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.861966 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pqn\" (UniqueName: \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.862085 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.870094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.863066 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.899055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.879020 4395 generic.go:356] "Generic (PLEG): container finished" podID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerID="4cff0fb6c98cc46f4f1ecfa39ff5bb696d580083432f295ac0975202e551fe03" exitCode=0 Nov 01 22:17:58.899055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.888580 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" event={"ID":"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4","Type":"ContainerDied","Data":"4cff0fb6c98cc46f4f1ecfa39ff5bb696d580083432f295ac0975202e551fe03"} Nov 01 22:17:58.918157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.916169 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pqn\" (UniqueName: \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\") pod \"payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:58.963782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.963748 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgpk\" (UniqueName: \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.963782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.963784 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.964943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.964915 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:58.983195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:58.983121 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgpk\" (UniqueName: \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\") pod \"payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:59.093970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.093938 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:17:59.145047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.145003 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:59.166951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.166758 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\" (UniqueName: \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\") pod \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " Nov 01 22:17:59.166951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.166909 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\") pod \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\" (UID: \"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4\") " Nov 01 22:17:59.167940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.167913 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume" pod "ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" (UID: "ea7dc798-11a9-4f2a-9314-7e7e4eca44c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:17:59.170587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.170563 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh" pod "ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" (UID: "ea7dc798-11a9-4f2a-9314-7e7e4eca44c4"). InnerVolumeSpecName "kube-api-access-7pklh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:17:59.223400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.223367 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:59.269005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.268973 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pklh\" (UniqueName: \"kubernetes.io/projected/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-kube-api-access-7pklh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:59.269005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.269002 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:17:59.446020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.445990 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:17:59.446187 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:17:59.446113 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca67a24e_6d5f_4aca_9941_dc702d7cd3ba.slice/crio-ed6ba829c911b4c0289b78d3125a8d4f6fdec5204eb6a38f614146c8db7f0adb WatchSource:0}: Error finding container ed6ba829c911b4c0289b78d3125a8d4f6fdec5204eb6a38f614146c8db7f0adb: Status 404 returned error can't find the container with id ed6ba829c911b4c0289b78d3125a8d4f6fdec5204eb6a38f614146c8db7f0adb Nov 01 22:17:59.543125 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:17:59.543095 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode991dc18_a776_41e8_83b9_a2b9b4af510f.slice/crio-7e3dbabf78ade69ebae4c768114ec636497b77f870e846b05da4fac859a1bb32 WatchSource:0}: Error finding container 7e3dbabf78ade69ebae4c768114ec636497b77f870e846b05da4fac859a1bb32: Status 404 returned error can't find the container with id 7e3dbabf78ade69ebae4c768114ec636497b77f870e846b05da4fac859a1bb32 Nov 01 22:17:59.543593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.543535 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:17:59.897756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.897653 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" event={"ID":"e991dc18-a776-41e8-83b9-a2b9b4af510f","Type":"ContainerStarted","Data":"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899"} Nov 01 22:17:59.897756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.897737 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:17:59.898414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.897764 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" event={"ID":"e991dc18-a776-41e8-83b9-a2b9b4af510f","Type":"ContainerStarted","Data":"7e3dbabf78ade69ebae4c768114ec636497b77f870e846b05da4fac859a1bb32"} Nov 01 22:17:59.899752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.899720 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" event={"ID":"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba","Type":"ContainerStarted","Data":"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1"} Nov 01 22:17:59.899867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.899769 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" event={"ID":"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba","Type":"ContainerStarted","Data":"ed6ba829c911b4c0289b78d3125a8d4f6fdec5204eb6a38f614146c8db7f0adb"} Nov 01 22:17:59.899867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.899798 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:17:59.901221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.901197 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" event={"ID":"ea7dc798-11a9-4f2a-9314-7e7e4eca44c4","Type":"ContainerDied","Data":"99018ac83d411f3f5593862f86b604701d173b36ba65d9feca904e798f20eddc"} Nov 01 22:17:59.901326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.901221 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np" Nov 01 22:17:59.901326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.901246 4395 scope.go:122] "RemoveContainer" containerID="4cff0fb6c98cc46f4f1ecfa39ff5bb696d580083432f295ac0975202e551fe03" Nov 01 22:17:59.920590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.920539 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" podStartSLOduration=1.920520975 podStartE2EDuration="1.920520975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:59.917864929 +0000 UTC m=+2026.891767467" watchObservedRunningTime="2026-11-01 22:17:59.920520975 +0000 UTC m=+2026.894423516" Nov 01 22:17:59.945260 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.945212 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" podStartSLOduration=1.9451951379999999 podStartE2EDuration="1.945195138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:59.943738645 +0000 UTC m=+2026.917641185" watchObservedRunningTime="2026-11-01 22:17:59.945195138 +0000 UTC m=+2026.919097678" Nov 01 22:17:59.962400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.962364 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:17:59.966231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:17:59.966204 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3e300c-6ddd474b99-hq8np"] Nov 01 22:18:01.167742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:01.167701 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" path="/var/lib/kubelet/pods/ea7dc798-11a9-4f2a-9314-7e7e4eca44c4/volumes" Nov 01 22:18:10.921649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:10.921557 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:18:10.926867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:10.926841 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:18:15.055222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.055186 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:18:15.056253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.056226 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" Nov 01 22:18:15.056253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.056250 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" Nov 01 22:18:15.056478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.056409 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7dc798-11a9-4f2a-9314-7e7e4eca44c4" containerName="payload-pre-processing" Nov 01 22:18:15.061696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.061664 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.326991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327093 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327246 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327280 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327301 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327384 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr795\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327455 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.327628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.327489 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.334867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.334840 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:18:15.341084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.341065 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:18:15.341265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.341251 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:18:15.341380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.341184 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.428801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.428768 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.428984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.428834 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.428984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.428880 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.428984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.428971 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429000 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429042 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chk8p\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.429195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429076 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429157 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429200 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429239 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr795\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429289 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429315 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429315 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429379 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.429816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429433 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.429816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.429513 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.430343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.430093 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.430343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.430167 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.430343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.430220 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.430343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.430238 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.430860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.430667 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.431858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.431822 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.432648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.432620 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.432864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.432845 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.434649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.434621 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.442447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.442325 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.442447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.442418 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr795\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.531409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531369 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531439 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531511 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531578 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531643 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531689 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531751 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531844 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531934 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chk8p\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.531997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.531943 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.532176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.532021 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.532363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.532326 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.532451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.532410 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.532451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.532413 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.533978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.533952 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.534151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.534134 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.576924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.576828 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.578406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.578299 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chk8p\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.663498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.663467 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:15.706826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.706795 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:15.956577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.956553 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:18:15.959063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.958984 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:15.959193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.959097 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:15.959193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:15.959147 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:16.014956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.014926 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:18:16.023506 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:18:16.023482 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f06aaa_bc7e_48bd_871d_9b09cd1efb37.slice/crio-439efe6b5a50707f4471c5d83f24fc125681b196291b8e4d4529ee01ccc954cf WatchSource:0}: Error finding container 439efe6b5a50707f4471c5d83f24fc125681b196291b8e4d4529ee01ccc954cf: Status 404 returned error can't find the container with id 439efe6b5a50707f4471c5d83f24fc125681b196291b8e4d4529ee01ccc954cf Nov 01 22:18:16.026795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.026761 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:16.026881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.026827 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:16.026881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.026859 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:16.195052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.194153 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" event={"ID":"b28ee681-0fd2-4f75-8cd4-5358aea3f50d","Type":"ContainerStarted","Data":"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920"} Nov 01 22:18:16.195052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.194223 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" event={"ID":"b28ee681-0fd2-4f75-8cd4-5358aea3f50d","Type":"ContainerStarted","Data":"6d5451f7c9c9310ca36ed8150a92b397b0003a8ac39d70257ea5a3f1b80f0f5c"} Nov 01 22:18:16.209106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.197267 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" event={"ID":"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37","Type":"ContainerStarted","Data":"f801bd48f2a73a39e5f11f3385209744bda6849b8e690bbeaf126c86055ce1e0"} Nov 01 22:18:16.209106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.197328 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" event={"ID":"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37","Type":"ContainerStarted","Data":"439efe6b5a50707f4471c5d83f24fc125681b196291b8e4d4529ee01ccc954cf"} Nov 01 22:18:16.219974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.219797 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podStartSLOduration=1.219780388 podStartE2EDuration="1.219780388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:16.219291224 +0000 UTC m=+2043.193193764" watchObservedRunningTime="2026-11-01 22:18:16.219780388 +0000 UTC m=+2043.193682926" Nov 01 22:18:16.247054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.246928 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podStartSLOduration=1.246910104 podStartE2EDuration="1.246910104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:16.241467679 +0000 UTC m=+2043.215370215" watchObservedRunningTime="2026-11-01 22:18:16.246910104 +0000 UTC m=+2043.220812626" Nov 01 22:18:16.664101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.664068 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:16.707688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:16.707652 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:17.673053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:17.672006 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:17.673053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:17.672094 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:17.714053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:17.713135 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:17.714053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:17.713200 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:18.664631 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:18.664483 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:18.664631 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:18.664530 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:18.723909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:18.714092 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:18.723909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:18.714150 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:19.664394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:19.664356 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:19.664581 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:19.664406 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:19.717052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:19.716415 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:19.717052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:19.716480 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:20.664391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:20.664357 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:20.664772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:20.664413 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:20.709052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:20.708618 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:20.709052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:20.708673 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:21.664238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:21.664205 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:21.664480 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:21.664454 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:21.714457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:21.714418 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:21.715111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:21.714696 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:22.415846 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:18:22.415807 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:18:22.664248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:22.664210 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 22:18:22.664451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:22.664271 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 22:18:22.707977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:22.707893 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:18:22.707977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:22.707952 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:18:23.669242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.669211 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:23.669726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.669704 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:23.670316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.670294 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:18:23.712188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.712167 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:23.712520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.712479 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:23.713167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:23.713151 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:18:30.192285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.191812 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:18:30.199121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.199091 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.271130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.216813 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.368955 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369007 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369092 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369130 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55bl\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369218 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369288 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369336 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369403 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.372055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.369484 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473208 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473250 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473286 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473338 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473461 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473476 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473509 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473533 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55bl\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.473648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.473597 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.474437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.474400 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.474641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.474392 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.475364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.475327 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.475524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.475501 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.475705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.475680 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.476861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.476842 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.478543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.478518 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.488140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.488106 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55bl\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.488452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.488431 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:30.756103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:30.753427 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:31.078557 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:18:31.078526 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12ec669_a26f_4b71_aeda_e00818e9421b.slice/crio-f8074d1544e5fa3129b715d0fe1c1eba959181a701167ad521d193b28ee7f97f WatchSource:0}: Error finding container f8074d1544e5fa3129b715d0fe1c1eba959181a701167ad521d193b28ee7f97f: Status 404 returned error can't find the container with id f8074d1544e5fa3129b715d0fe1c1eba959181a701167ad521d193b28ee7f97f Nov 01 22:18:31.079813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.079786 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:18:31.081819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.081784 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:31.081907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.081866 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:31.081950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.081907 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:31.491917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.491817 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" event={"ID":"b12ec669-a26f-4b71-aeda-e00818e9421b","Type":"ContainerStarted","Data":"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1"} Nov 01 22:18:31.492590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.492565 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" event={"ID":"b12ec669-a26f-4b71-aeda-e00818e9421b","Type":"ContainerStarted","Data":"f8074d1544e5fa3129b715d0fe1c1eba959181a701167ad521d193b28ee7f97f"} Nov 01 22:18:31.538677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.537059 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podStartSLOduration=1.537017583 podStartE2EDuration="1.537017583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:31.525774444 +0000 UTC m=+2058.499676982" watchObservedRunningTime="2026-11-01 22:18:31.537017583 +0000 UTC m=+2058.510920121" Nov 01 22:18:31.754525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:31.754493 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:32.755753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:32.755712 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:18:32.756185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:32.755768 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 01 22:18:33.754803 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:33.754766 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:18:33.755098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:33.754833 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 01 22:18:34.754736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:34.754702 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:18:34.755182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:34.754759 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 01 22:18:35.764056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:35.763403 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:18:35.764056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:35.763464 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 01 22:18:36.754742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:36.754699 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 01 22:18:36.754934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:36.754788 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 01 22:18:37.895350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:37.895313 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:37.896342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:37.896317 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:37.897150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:37.897130 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:18:45.007896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.007862 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:18:45.013467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.013443 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.018898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.018872 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-e2bf63-openshift-default-dockercfg-tjbqw\"" Nov 01 22:18:45.050142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.050108 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:18:45.080793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.080753 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.080999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.080809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081022 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081090 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081137 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081198 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081268 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081296 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.081466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.081383 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfrk\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183104 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183139 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183177 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183199 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183286 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183323 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183361 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfrk\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183407 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183433 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183529 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183788 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.183909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183885 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.184004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.183980 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.184347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.184283 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.186003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.185980 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.186185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.186139 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.195764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.195737 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.196250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.196227 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfrk\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.336337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.336270 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:45.633155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.633124 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:18:45.633803 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:18:45.633779 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168a20e7_07dd_49e4_a8ce_64fcbf16938f.slice/crio-7e4cc196360fe3ed0d588b4379928cd724d83ba879352faf4c9645513e9d0e54 WatchSource:0}: Error finding container 7e4cc196360fe3ed0d588b4379928cd724d83ba879352faf4c9645513e9d0e54: Status 404 returned error can't find the container with id 7e4cc196360fe3ed0d588b4379928cd724d83ba879352faf4c9645513e9d0e54 Nov 01 22:18:45.636798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.636766 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:45.636875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.636833 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:45.636875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.636863 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:18:45.686014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:45.685988 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" event={"ID":"168a20e7-07dd-49e4-a8ce-64fcbf16938f","Type":"ContainerStarted","Data":"7e4cc196360fe3ed0d588b4379928cd724d83ba879352faf4c9645513e9d0e54"} Nov 01 22:18:46.695999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:46.695954 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" event={"ID":"168a20e7-07dd-49e4-a8ce-64fcbf16938f","Type":"ContainerStarted","Data":"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa"} Nov 01 22:18:46.724182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:46.724122 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podStartSLOduration=2.724103161 podStartE2EDuration="2.724103161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:46.720243842 +0000 UTC m=+2073.694146382" watchObservedRunningTime="2026-11-01 22:18:46.724103161 +0000 UTC m=+2073.698005700" Nov 01 22:18:47.337259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:47.337222 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:47.338663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:47.338638 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:47.338795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:47.338695 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:48.336853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:48.336805 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:48.337313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:48.336888 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:49.342497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:49.342453 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:49.343171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:49.343137 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:50.337021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:50.336986 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:50.337276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:50.337070 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:51.337101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:51.337020 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:51.337520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:51.337167 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:52.305670 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:18:52.305628 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:18:52.336690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:52.336654 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:18:52.336852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:52.336725 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 22:18:53.343308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:53.343277 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:53.343834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:53.343813 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:18:53.344848 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:18:53.344825 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:19:21.943432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:21.943400 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:19:22.306982 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:19:22.306937 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:19:33.390410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:33.390317 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:19:33.390924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:33.390558 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" podUID="d74a83d8-861c-42fc-959e-f83751c4758f" containerName="maas-api" containerID="cri-o://520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89" gracePeriod=30 Nov 01 22:19:33.954721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:33.954691 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:19:34.115649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.115606 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\" (UniqueName: \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\") pod \"d74a83d8-861c-42fc-959e-f83751c4758f\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " Nov 01 22:19:34.116073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.116048 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\") pod \"d74a83d8-861c-42fc-959e-f83751c4758f\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " Nov 01 22:19:34.116399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.116373 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\") pod \"d74a83d8-861c-42fc-959e-f83751c4758f\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " Nov 01 22:19:34.116656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.116637 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\") pod \"d74a83d8-861c-42fc-959e-f83751c4758f\" (UID: \"d74a83d8-861c-42fc-959e-f83751c4758f\") " Nov 01 22:19:34.132053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.130271 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca" pod "d74a83d8-861c-42fc-959e-f83751c4758f" (UID: "d74a83d8-861c-42fc-959e-f83751c4758f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:34.153817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.153658 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk" pod "d74a83d8-861c-42fc-959e-f83751c4758f" (UID: "d74a83d8-861c-42fc-959e-f83751c4758f"). InnerVolumeSpecName "kube-api-access-rc2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:34.167659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.167563 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs" pod "d74a83d8-861c-42fc-959e-f83751c4758f" (UID: "d74a83d8-861c-42fc-959e-f83751c4758f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:34.170194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.170162 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls" pod "d74a83d8-861c-42fc-959e-f83751c4758f" (UID: "d74a83d8-861c-42fc-959e-f83751c4758f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:34.219008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.218979 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:34.219008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.219006 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2sk\" (UniqueName: \"kubernetes.io/projected/d74a83d8-861c-42fc-959e-f83751c4758f-kube-api-access-rc2sk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:34.219215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.219017 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74a83d8-861c-42fc-959e-f83751c4758f-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:34.219215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.219042 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74a83d8-861c-42fc-959e-f83751c4758f-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:34.336450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.336419 4395 generic.go:356] "Generic (PLEG): container finished" podID="d74a83d8-861c-42fc-959e-f83751c4758f" containerID="520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89" exitCode=0 Nov 01 22:19:34.336647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.336475 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" Nov 01 22:19:34.336647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.336505 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" event={"ID":"d74a83d8-861c-42fc-959e-f83751c4758f","Type":"ContainerDied","Data":"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89"} Nov 01 22:19:34.336647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.336569 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc" event={"ID":"d74a83d8-861c-42fc-959e-f83751c4758f","Type":"ContainerDied","Data":"e215ac2ba09cdf35e5a7ccaa3c372f42d23d3d57462bcdc167cdd43f440dbfb2"} Nov 01 22:19:34.336647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.336599 4395 scope.go:122] "RemoveContainer" containerID="520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89" Nov 01 22:19:34.348269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.348253 4395 scope.go:122] "RemoveContainer" containerID="520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89" Nov 01 22:19:34.348570 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:19:34.348535 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89\": container with ID starting with 520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89 not found: ID does not exist" containerID="520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89" Nov 01 22:19:34.348639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.348565 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89"} err="failed to get container status \"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89\": rpc error: code = NotFound desc = could not find container \"520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89\": container with ID starting with 520d3e7cfc1f0beed42c7ae27142afa80dcc9df994371da0bf47c92bb4d47a89 not found: ID does not exist" Nov 01 22:19:34.371503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.371476 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:19:34.376574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:34.376554 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dfdb11-74cf8f755d-bg5tc"] Nov 01 22:19:35.167514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:35.167486 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74a83d8-861c-42fc-959e-f83751c4758f" path="/var/lib/kubelet/pods/d74a83d8-861c-42fc-959e-f83751c4758f/volumes" Nov 01 22:19:36.980054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:36.979685 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" containerID="cri-o://17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828" gracePeriod=45 Nov 01 22:19:37.338912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.338885 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:19:37.372258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.372227 4395 generic.go:356] "Generic (PLEG): container finished" podID="4d30a263-029d-465f-ba01-5de67184fa12" containerID="17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828" exitCode=0 Nov 01 22:19:37.372421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.372280 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" Nov 01 22:19:37.372421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.372316 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" event={"ID":"4d30a263-029d-465f-ba01-5de67184fa12","Type":"ContainerDied","Data":"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828"} Nov 01 22:19:37.372421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.372374 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" event={"ID":"4d30a263-029d-465f-ba01-5de67184fa12","Type":"ContainerDied","Data":"243195cd8a474af14de8d3a6fefb0669989e84159c76f6946f9e3d344396ac14"} Nov 01 22:19:37.372421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.372401 4395 scope.go:122] "RemoveContainer" containerID="17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828" Nov 01 22:19:37.385091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.385061 4395 scope.go:122] "RemoveContainer" containerID="17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828" Nov 01 22:19:37.385390 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:19:37.385371 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828\": container with ID starting with 17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828 not found: ID does not exist" containerID="17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828" Nov 01 22:19:37.385478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.385397 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828"} err="failed to get container status \"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828\": rpc error: code = NotFound desc = could not find container \"17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828\": container with ID starting with 17fc40a628d66dd9785f6a7e10d4bf1f03a4a8f554353791d62ad1f607bbd828 not found: ID does not exist" Nov 01 22:19:37.487123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487096 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487205 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487318 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487379 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487489 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:37.487639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487543 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487566 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:37.487639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487608 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\" (UniqueName: \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\") pod \"4d30a263-029d-465f-ba01-5de67184fa12\" (UID: \"4d30a263-029d-465f-ba01-5de67184fa12\") " Nov 01 22:19:37.487807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.487777 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:37.488277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.488207 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.488277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.488227 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.488277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.488265 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.489487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.489464 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:19:37.489792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.489776 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:37.490306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.490281 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2" pod "4d30a263-029d-465f-ba01-5de67184fa12" (UID: "4d30a263-029d-465f-ba01-5de67184fa12"). InnerVolumeSpecName "kube-api-access-t5px2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:37.589903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.589840 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5px2\" (UniqueName: \"kubernetes.io/projected/4d30a263-029d-465f-ba01-5de67184fa12-kube-api-access-t5px2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.589903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.589869 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d30a263-029d-465f-ba01-5de67184fa12-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.589903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.589883 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d30a263-029d-465f-ba01-5de67184fa12-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:37.701578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.701556 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:19:37.706416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:37.706397 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8"] Nov 01 22:19:38.213546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:38.213506 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-070e3d/test-model-070e3d-kserve-77fcddbf7c-gjqh8" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:39.170992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:39.170951 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d30a263-029d-465f-ba01-5de67184fa12" path="/var/lib/kubelet/pods/4d30a263-029d-465f-ba01-5de67184fa12/volumes" Nov 01 22:19:41.219608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:41.219563 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:19:41.220124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:41.219866 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" containerID="cri-o://19d01e68bef4624cc77c2ce739439dece113385f7512359e28c90285fcfdf13b" gracePeriod=30 Nov 01 22:19:44.027690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:44.027650 4395 patch_prober.go:29] interesting pod/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:19:44.028242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:44.027715 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:19:46.516078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.512910 4395 generic.go:356] "Generic (PLEG): container finished" podID="698859d6-5084-4b99-8d80-610441e98fea" containerID="19d01e68bef4624cc77c2ce739439dece113385f7512359e28c90285fcfdf13b" exitCode=0 Nov 01 22:19:46.516078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.513117 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" event={"ID":"698859d6-5084-4b99-8d80-610441e98fea","Type":"ContainerDied","Data":"19d01e68bef4624cc77c2ce739439dece113385f7512359e28c90285fcfdf13b"} Nov 01 22:19:46.603243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.603218 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:19:46.709884 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.709849 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.709896 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.709930 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.709948 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.709977 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710129 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:46.710350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710147 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710230 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710297 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710376 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:46.710578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710447 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\") pod \"698859d6-5084-4b99-8d80-610441e98fea\" (UID: \"698859d6-5084-4b99-8d80-610441e98fea\") " Nov 01 22:19:46.710896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710682 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:46.711100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.710913 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:19:46.711241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.711168 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.711241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.711191 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.711369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.711302 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:46.712816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.712715 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:46.712816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.712793 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:19:46.712961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.712883 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "kube-api-access-wc5cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:19:46.712961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.712900 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy" pod "698859d6-5084-4b99-8d80-610441e98fea" (UID: "698859d6-5084-4b99-8d80-610441e98fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:19:46.812460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812410 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812435 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812444 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698859d6-5084-4b99-8d80-610441e98fea-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812453 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698859d6-5084-4b99-8d80-610441e98fea-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812462 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5cj\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-kube-api-access-wc5cj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812472 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698859d6-5084-4b99-8d80-610441e98fea-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:46.812659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:46.812480 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698859d6-5084-4b99-8d80-610441e98fea-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:19:47.521667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:47.521635 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" event={"ID":"698859d6-5084-4b99-8d80-610441e98fea","Type":"ContainerDied","Data":"8031237a33ca6961c1225abf72243ad960d0ea0c97207587e66486b68db6069b"} Nov 01 22:19:47.522135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:47.521699 4395 scope.go:122] "RemoveContainer" containerID="19d01e68bef4624cc77c2ce739439dece113385f7512359e28c90285fcfdf13b" Nov 01 22:19:47.522135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:47.521698 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55" Nov 01 22:19:47.548219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:47.548190 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:19:47.558448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:47.558417 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dfdb11-openshift-default-75fbd6bd67-ldt55"] Nov 01 22:19:49.167167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:49.167137 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698859d6-5084-4b99-8d80-610441e98fea" path="/var/lib/kubelet/pods/698859d6-5084-4b99-8d80-610441e98fea/volumes" Nov 01 22:19:51.899254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:51.899220 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:19:52.520107 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:19:52.520075 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:19:56.900164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:56.900110 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" containerID="cri-o://d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d" gracePeriod=25 Nov 01 22:19:57.799234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:57.799187 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:19:58.363169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:19:58.363127 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:20:05.799460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:05.799420 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:20:07.659422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:07.659384 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:20:07.659918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:07.659658 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" containerID="cri-o://4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674" gracePeriod=30 Nov 01 22:20:09.026864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:09.026831 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2"] Nov 01 22:20:09.030932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:09.030902 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a791869f-z5cx2"] Nov 01 22:20:09.251326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:09.251290 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d" path="/var/lib/kubelet/pods/2f5be401-9cd6-4f3c-a6c7-0dcfa2cfc40d/volumes" Nov 01 22:20:12.989628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:12.989603 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:20:13.056060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.055636 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.056060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.055715 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.056060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.055781 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.056060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.055838 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.056060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.055884 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.058158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.057554 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.058158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.057917 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.058158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.058045 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:20:13.058700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.058433 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.058700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.058485 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.058928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.058696 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.059059 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.059134 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\") pod \"7639c72f-460d-42d2-a9c6-e5cc6b767400\" (UID: \"7639c72f-460d-42d2-a9c6-e5cc6b767400\") " Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.059336 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060063 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060084 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7639c72f-460d-42d2-a9c6-e5cc6b767400-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060099 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060113 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.060150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060127 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.061076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.060996 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:20:13.066324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.066299 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:20:13.066515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.066490 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:13.067945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.067905 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx" pod "7639c72f-460d-42d2-a9c6-e5cc6b767400" (UID: "7639c72f-460d-42d2-a9c6-e5cc6b767400"). InnerVolumeSpecName "kube-api-access-xk2dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:13.161064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.161018 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.161064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.161058 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.161064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.161067 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7639c72f-460d-42d2-a9c6-e5cc6b767400-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.161302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.161096 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2dx\" (UniqueName: \"kubernetes.io/projected/7639c72f-460d-42d2-a9c6-e5cc6b767400-kube-api-access-xk2dx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:13.276924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.276886 4395 generic.go:356] "Generic (PLEG): container finished" podID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerID="4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674" exitCode=0 Nov 01 22:20:13.277098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.276926 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" Nov 01 22:20:13.277098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.276966 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" event={"ID":"7639c72f-460d-42d2-a9c6-e5cc6b767400","Type":"ContainerDied","Data":"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674"} Nov 01 22:20:13.277098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.277039 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k" event={"ID":"7639c72f-460d-42d2-a9c6-e5cc6b767400","Type":"ContainerDied","Data":"55d833fc5f27dfa7e55d6d6f2eb5b84828e670964107b6c9278101746437b6ed"} Nov 01 22:20:13.277098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.277070 4395 scope.go:122] "RemoveContainer" containerID="4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674" Nov 01 22:20:13.288417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.288393 4395 scope.go:122] "RemoveContainer" containerID="4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674" Nov 01 22:20:13.288677 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:20:13.288650 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674\": container with ID starting with 4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674 not found: ID does not exist" containerID="4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674" Nov 01 22:20:13.288740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.288682 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674"} err="failed to get container status \"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674\": rpc error: code = NotFound desc = could not find container \"4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674\": container with ID starting with 4d70ba94ffe4e6a8ceab4aecc0801adb5c312df4a7d76f19c5d0ed8ba283e674 not found: ID does not exist" Nov 01 22:20:13.300454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.300432 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:20:13.304264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:13.304243 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-070e3d-openshift-default-68c88bd895-hqx8k"] Nov 01 22:20:15.171050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.169564 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" path="/var/lib/kubelet/pods/7639c72f-460d-42d2-a9c6-e5cc6b767400/volumes" Nov 01 22:20:15.798574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.798536 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:20:15.845778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.845742 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:20:15.846474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846460 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" Nov 01 22:20:15.846527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846475 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" Nov 01 22:20:15.846527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846488 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" Nov 01 22:20:15.846527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846493 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846533 4395 cpu_manager.go:414] "containerMap: removing container" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846539 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846547 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d74a83d8-861c-42fc-959e-f83751c4758f" containerName="maas-api" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846551 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a83d8-861c-42fc-959e-f83751c4758f" containerName="maas-api" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846651 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7639c72f-460d-42d2-a9c6-e5cc6b767400" containerName="istio-proxy" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846660 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d30a263-029d-465f-ba01-5de67184fa12" containerName="main" Nov 01 22:20:15.846670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846672 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="698859d6-5084-4b99-8d80-610441e98fea" containerName="istio-proxy" Nov 01 22:20:15.846907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.846678 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74a83d8-861c-42fc-959e-f83751c4758f" containerName="maas-api" Nov 01 22:20:15.852078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.852058 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.858253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.858221 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d6e25f-openshift-default-dockercfg-t9xvc\"" Nov 01 22:20:15.863531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.863508 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:20:15.942159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942128 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942168 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvldh\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942196 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942456 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942509 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942615 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942719 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942784 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:15.942950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:15.942931 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.045751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.045705 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.045963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.045821 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.045963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.045893 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.046170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.045971 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.046273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.046182 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.046409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.046390 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.046469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.046439 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvldh\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.046546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.046483 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.051959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.050898 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.052454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.052303 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.052992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.052665 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.053627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.053218 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.053627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.053280 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.053627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.053472 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.058183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.058157 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.058499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.058382 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.064315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.064296 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.065194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.065172 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvldh\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.168252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.168231 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:16.469437 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:20:16.469405 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39894a71_8d71_4cab_bbcb_2ec5aeceed9f.slice/crio-e796f324fd46a81ab3e7615895a6546741cef253bd1e39154bd93d341811f02d WatchSource:0}: Error finding container e796f324fd46a81ab3e7615895a6546741cef253bd1e39154bd93d341811f02d: Status 404 returned error can't find the container with id e796f324fd46a81ab3e7615895a6546741cef253bd1e39154bd93d341811f02d Nov 01 22:20:16.469819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.469794 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:20:16.472204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.472168 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:20:16.472305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.472239 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:20:16.472305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:16.472266 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:20:17.108696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.108674 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:20:17.180471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.180278 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\") pod \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " Nov 01 22:20:17.187231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.180491 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\" (UniqueName: \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\") pod \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\" (UID: \"6396d252-a034-4f8e-b7c4-7d88278c2ef6\") " Nov 01 22:20:17.187231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.180900 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume" pod "6396d252-a034-4f8e-b7c4-7d88278c2ef6" (UID: "6396d252-a034-4f8e-b7c4-7d88278c2ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:20:17.187231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.181581 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6396d252-a034-4f8e-b7c4-7d88278c2ef6-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:17.187231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.184274 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r" pod "6396d252-a034-4f8e-b7c4-7d88278c2ef6" (UID: "6396d252-a034-4f8e-b7c4-7d88278c2ef6"). InnerVolumeSpecName "kube-api-access-s2g8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:20:17.282466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.282441 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g8r\" (UniqueName: \"kubernetes.io/projected/6396d252-a034-4f8e-b7c4-7d88278c2ef6-kube-api-access-s2g8r\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:20:17.328425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.328390 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" event={"ID":"39894a71-8d71-4cab-bbcb-2ec5aeceed9f","Type":"ContainerStarted","Data":"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d"} Nov 01 22:20:17.328593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.328453 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" event={"ID":"39894a71-8d71-4cab-bbcb-2ec5aeceed9f","Type":"ContainerStarted","Data":"e796f324fd46a81ab3e7615895a6546741cef253bd1e39154bd93d341811f02d"} Nov 01 22:20:17.330103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.330071 4395 generic.go:356] "Generic (PLEG): container finished" podID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerID="d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d" exitCode=0 Nov 01 22:20:17.330247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.330120 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" Nov 01 22:20:17.330247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.330121 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" event={"ID":"6396d252-a034-4f8e-b7c4-7d88278c2ef6","Type":"ContainerDied","Data":"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d"} Nov 01 22:20:17.330247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.330239 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9" event={"ID":"6396d252-a034-4f8e-b7c4-7d88278c2ef6","Type":"ContainerDied","Data":"1a75927e70de6c852df8d936394757c35cddd9c6428c045d6a07566edf1e46d9"} Nov 01 22:20:17.330424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.330277 4395 scope.go:122] "RemoveContainer" containerID="d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d" Nov 01 22:20:17.342213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.342196 4395 scope.go:122] "RemoveContainer" containerID="d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d" Nov 01 22:20:17.342489 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:20:17.342472 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d\": container with ID starting with d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d not found: ID does not exist" containerID="d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d" Nov 01 22:20:17.342542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.342496 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d"} err="failed to get container status \"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d\": rpc error: code = NotFound desc = could not find container \"d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d\": container with ID starting with d24c22011948609b0f676254df97b2cd9481d9306612a976237da6447e48645d not found: ID does not exist" Nov 01 22:20:17.353989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.353943 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podStartSLOduration=2.353928137 podStartE2EDuration="2.353928137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:20:17.350408713 +0000 UTC m=+2164.324311251" watchObservedRunningTime="2026-11-01 22:20:17.353928137 +0000 UTC m=+2164.327830676" Nov 01 22:20:17.370229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.370207 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:20:17.375852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:17.375826 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-070e3d-5448b9d84c-t6vp9"] Nov 01 22:20:18.169373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:18.169342 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:18.171177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:18.171147 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:18.171341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:18.171200 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:19.025277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.025247 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b"] Nov 01 22:20:19.029210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.029185 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ddd222f-9nv5b"] Nov 01 22:20:19.169189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.169154 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:19.169373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.169204 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:19.170673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.170650 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" path="/var/lib/kubelet/pods/6396d252-a034-4f8e-b7c4-7d88278c2ef6/volumes" Nov 01 22:20:19.171194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:19.171179 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa611729-a77b-412b-9468-11557397eb9d" path="/var/lib/kubelet/pods/aa611729-a77b-412b-9468-11557397eb9d/volumes" Nov 01 22:20:20.169308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:20.169271 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:20.169482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:20.169343 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:21.168853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:21.168820 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:21.169276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:21.168870 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:22.169510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:22.169476 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:22.169912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:22.169528 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:22.404064 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:20:22.402303 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:20:23.207095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:23.206038 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:20:23.207095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:23.206116 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:20:24.173560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:24.173529 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:24.174009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:24.173988 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:24.174724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:20:24.174705 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:20:52.304914 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:20:52.304860 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:21:04.320066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.319823 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:21:04.321818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.321785 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" Nov 01 22:21:04.330460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.330428 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" Nov 01 22:21:04.331079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.331057 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6396d252-a034-4f8e-b7c4-7d88278c2ef6" containerName="payload-pre-processing" Nov 01 22:21:04.336256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.336160 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:21:04.336418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.336348 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.365144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.364239 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:21:04.365144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.364373 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.365530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.365502 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.365770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.365740 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.365952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.365932 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.366322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.366303 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.366504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.366487 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864fk\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.366666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.366651 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.366798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.366782 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.366928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.366914 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.367057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.367041 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.376608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.376569 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:21:04.430145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.430112 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-521734-openshift-default-dockercfg-wmbjz\"" Nov 01 22:21:04.469023 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.468985 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469134 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469190 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469222 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864fk\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469300 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469360 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469414 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469471 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469576 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469636 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469671 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vvg\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469692 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469784 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469829 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469853 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.469878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469860 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.469985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.469968 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.470141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.470122 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.470183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.470133 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.470217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.470179 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.470351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.470331 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.470391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.470371 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.471972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.471944 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.472759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.472736 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.482831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.482805 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864fk\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.483344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.483325 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.572533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572429 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.572533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572485 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vvg\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.572790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572548 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.572790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572673 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.572790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572723 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.572790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572767 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572916 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572961 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.572968 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.573173 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.573207 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.573276 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.573283 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.573514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.573496 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.575336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.575313 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.575596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.575579 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.585131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.585099 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.585316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.585242 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vvg\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.684643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.684610 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:04.698284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.698261 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:04.980936 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:21:04.980903 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a14cb7_9703_416a_9730_83950bb34b08.slice/crio-48a5f9053c51eba6d3f82b1b0b6f2bb63f033eec99af4946befab8553fa37966 WatchSource:0}: Error finding container 48a5f9053c51eba6d3f82b1b0b6f2bb63f033eec99af4946befab8553fa37966: Status 404 returned error can't find the container with id 48a5f9053c51eba6d3f82b1b0b6f2bb63f033eec99af4946befab8553fa37966 Nov 01 22:21:04.981638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.981612 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:21:04.984007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.983979 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:04.984118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.984086 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:04.984118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.984114 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:04.996929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:04.996901 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:21:04.997895 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:21:04.997867 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31968d07_cb32_46ad_a888_d979e624c5e0.slice/crio-b52aa9747dbbc63b7a9f6f36b7e399f4bf759b637c24d2dffb08157b54a1108f WatchSource:0}: Error finding container b52aa9747dbbc63b7a9f6f36b7e399f4bf759b637c24d2dffb08157b54a1108f: Status 404 returned error can't find the container with id b52aa9747dbbc63b7a9f6f36b7e399f4bf759b637c24d2dffb08157b54a1108f Nov 01 22:21:05.001021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.000990 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:05.001137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.001123 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:05.001190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.001172 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:05.840916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.840867 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" event={"ID":"60a14cb7-9703-416a-9730-83950bb34b08","Type":"ContainerStarted","Data":"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108"} Nov 01 22:21:05.841467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.841253 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" event={"ID":"60a14cb7-9703-416a-9730-83950bb34b08","Type":"ContainerStarted","Data":"48a5f9053c51eba6d3f82b1b0b6f2bb63f033eec99af4946befab8553fa37966"} Nov 01 22:21:05.843170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.843143 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" event={"ID":"31968d07-cb32-46ad-a888-d979e624c5e0","Type":"ContainerStarted","Data":"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3"} Nov 01 22:21:05.843286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.843185 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" event={"ID":"31968d07-cb32-46ad-a888-d979e624c5e0","Type":"ContainerStarted","Data":"b52aa9747dbbc63b7a9f6f36b7e399f4bf759b637c24d2dffb08157b54a1108f"} Nov 01 22:21:05.872053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.869668 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podStartSLOduration=1.869644176 podStartE2EDuration="1.869644176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:05.863751173 +0000 UTC m=+2212.837653712" watchObservedRunningTime="2026-11-01 22:21:05.869644176 +0000 UTC m=+2212.843546717" Nov 01 22:21:05.976070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:05.975825 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podStartSLOduration=1.97580033 podStartE2EDuration="1.97580033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:05.891538695 +0000 UTC m=+2212.865441462" watchObservedRunningTime="2026-11-01 22:21:05.97580033 +0000 UTC m=+2212.949702870" Nov 01 22:21:06.690056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.689533 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:06.690056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.689692 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:06.690056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.689744 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:06.708061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.698780 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:06.708061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.705327 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:06.708061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:06.705382 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:07.702056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:07.701440 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:07.702056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:07.701523 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:07.702682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:07.702382 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:07.702682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:07.702425 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:08.709055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:08.705455 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:08.709055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:08.705525 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:08.709055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:08.706992 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:08.709055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:08.707056 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:09.698049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:09.695407 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:09.698049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:09.695473 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:09.707061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:09.702983 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:09.707061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:09.703058 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:10.690084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:10.686410 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:10.690084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:10.686480 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:10.708057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:10.701211 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:10.708057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:10.701287 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:11.698056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:11.690290 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:21:11.698056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:11.690360 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:21:11.715842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:11.715805 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:21:11.716002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:11.715864 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:21:12.690511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.690481 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:12.690787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.690766 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:12.691388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.691369 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:21:12.712345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.712314 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:12.712747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.712576 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:12.713232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:12.713215 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:21:16.091874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:16.091842 4395 scope.go:122] "RemoveContainer" containerID="cb969808f0ed5ebc86981fc20ef9d063ddd7ccf928171e69992ebf26908f5a57" Nov 01 22:21:16.102481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:16.102459 4395 scope.go:122] "RemoveContainer" containerID="f0ebd277c69961953d62e6dede4c6c366df22c33e05b498e013e0f0cf6c539c3" Nov 01 22:21:22.303651 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:21:22.303609 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:21:49.194804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.194775 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 22:21:49.352981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.352935 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" containerID="cri-o://40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309" gracePeriod=30 Nov 01 22:21:49.364977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.364946 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:21:49.371684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.371661 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.404243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.404215 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:21:49.493487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493457 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493572 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493645 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493680 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493844 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493912 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndpk\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.493973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493960 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.494009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.493983 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.494218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.494197 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.596175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.596138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.596362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.596308 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.596445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.596373 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.597474 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.597656 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.598407 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.598459 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.598533 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.598582 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndpk\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.598613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.598610 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.599972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.599947 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.600483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.599955 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.600751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.600728 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.600938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.599554 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.606869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.606846 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.609630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.609399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.609630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.609598 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.612116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.612097 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndpk\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.646010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.645989 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 22:21:49.702364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702334 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.702549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702391 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.702549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702459 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.702674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702587 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.702674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702664 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.703079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702830 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.703079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.702887 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\") pod \"e39994c2-9b0f-4784-a9c6-16af42387eb3\" (UID: \"e39994c2-9b0f-4784-a9c6-16af42387eb3\") " Nov 01 22:21:49.704161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.703370 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:21:49.704161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.703774 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-ca-configmap\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.705819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.705784 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:49.709422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.709393 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:21:49.710380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.710316 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:21:49.710380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.710342 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:21:49.711193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.711169 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:49.712274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.712250 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk" pod "e39994c2-9b0f-4784-a9c6-16af42387eb3" (UID: "e39994c2-9b0f-4784-a9c6-16af42387eb3"). InnerVolumeSpecName "kube-api-access-vq6pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:49.776083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.775001 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:49.805064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805003 4395 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-cacerts\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.805184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805095 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-csr-dns-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.805184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805108 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.805184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805117 4395 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e39994c2-9b0f-4784-a9c6-16af42387eb3-istio-kubeconfig\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.805184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805125 4395 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39994c2-9b0f-4784-a9c6-16af42387eb3-local-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:49.805184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:49.805134 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6pk\" (UniqueName: \"kubernetes.io/projected/e39994c2-9b0f-4784-a9c6-16af42387eb3-kube-api-access-vq6pk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:50.077761 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:21:50.077732 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6e0519_0259_4f2b_bb4e_9179ae59f022.slice/crio-42cbbde1e79a385c1066320c93571eac482db45b956b123f85f1eeba5ba44bba WatchSource:0}: Error finding container 42cbbde1e79a385c1066320c93571eac482db45b956b123f85f1eeba5ba44bba: Status 404 returned error can't find the container with id 42cbbde1e79a385c1066320c93571eac482db45b956b123f85f1eeba5ba44bba Nov 01 22:21:50.078937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.078910 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:21:50.081290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.081260 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:50.081369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.081328 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:50.081369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.081356 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:21:50.364937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.364502 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" event={"ID":"de6e0519-0259-4f2b-bb4e-9179ae59f022","Type":"ContainerStarted","Data":"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d"} Nov 01 22:21:50.365491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.365242 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" event={"ID":"de6e0519-0259-4f2b-bb4e-9179ae59f022","Type":"ContainerStarted","Data":"42cbbde1e79a385c1066320c93571eac482db45b956b123f85f1eeba5ba44bba"} Nov 01 22:21:50.367272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.367230 4395 generic.go:356] "Generic (PLEG): container finished" podID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerID="40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309" exitCode=0 Nov 01 22:21:50.367429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.367311 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" Nov 01 22:21:50.367429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.367362 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" event={"ID":"e39994c2-9b0f-4784-a9c6-16af42387eb3","Type":"ContainerDied","Data":"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309"} Nov 01 22:21:50.367429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.367404 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" event={"ID":"e39994c2-9b0f-4784-a9c6-16af42387eb3","Type":"ContainerDied","Data":"cf944ea189f45fe46f6b0aa99d61751f14268d3e7cd5596f745fd672bded04d8"} Nov 01 22:21:50.367635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.367430 4395 scope.go:122] "RemoveContainer" containerID="40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309" Nov 01 22:21:50.371897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.371876 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 22:21:50.372295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.372247 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" containerID="cri-o://26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275" gracePeriod=30 Nov 01 22:21:50.416490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.416470 4395 scope.go:122] "RemoveContainer" containerID="40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309" Nov 01 22:21:50.416872 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:21:50.416842 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309\": container with ID starting with 40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309 not found: ID does not exist" containerID="40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309" Nov 01 22:21:50.416976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.416887 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309"} err="failed to get container status \"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309\": rpc error: code = NotFound desc = could not find container \"40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309\": container with ID starting with 40dc6684de9479f992c80611f3af1d10fa06c9dbced4bcbb3db6cea8e107b309 not found: ID does not exist" Nov 01 22:21:50.417214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.417169 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podStartSLOduration=1.417151222 podStartE2EDuration="1.417151222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:50.409297437 +0000 UTC m=+2257.383199989" watchObservedRunningTime="2026-11-01 22:21:50.417151222 +0000 UTC m=+2257.391053762" Nov 01 22:21:50.452888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.452859 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 22:21:50.472271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.472240 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd"] Nov 01 22:21:50.775759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:50.775731 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:51.169557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:51.169467 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" path="/var/lib/kubelet/pods/e39994c2-9b0f-4784-a9c6-16af42387eb3/volumes" Nov 01 22:21:51.778172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:51.776767 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:51.778172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:51.776829 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:52.327351 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:21:52.327312 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:21:52.781385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:52.781352 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:52.796134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:52.796092 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:53.796053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:53.795333 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:53.796053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:53.795393 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:54.562643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:54.562593 4395 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-782nd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:21:54.562979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:54.562680 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-782nd" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.93:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:21:54.792504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:54.792463 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:54.792826 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:54.792801 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:55.775825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.775788 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:55.776859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.776412 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:55.860915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.860887 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.981596 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.981660 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.981691 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.981771 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.981860 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.982057 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.982138 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.982203 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.987157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.982256 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\") pod \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\" (UID: \"059a38f7-1f72-418d-9c8a-fa3e86a8674c\") " Nov 01 22:21:55.988233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.988121 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:55.988422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.988397 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:55.988613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.988590 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:55.988805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:55.988782 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:56.012360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.012315 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:21:56.102332 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.102288 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127119 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127164 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127179 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127193 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127207 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.206053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.127220 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.291745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.291664 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:21:56.403061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.399868 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.408327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.408259 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:21:56.473058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.472539 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk" pod "059a38f7-1f72-418d-9c8a-fa3e86a8674c" (UID: "059a38f7-1f72-418d-9c8a-fa3e86a8674c"). InnerVolumeSpecName "kube-api-access-5j6tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:21:56.507052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.503135 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059a38f7-1f72-418d-9c8a-fa3e86a8674c-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.521123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.521019 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j6tk\" (UniqueName: \"kubernetes.io/projected/059a38f7-1f72-418d-9c8a-fa3e86a8674c-kube-api-access-5j6tk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:21:56.726358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.726322 4395 generic.go:356] "Generic (PLEG): container finished" podID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerID="26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275" exitCode=0 Nov 01 22:21:56.726853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.726827 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerDied","Data":"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275"} Nov 01 22:21:56.727117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.727094 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" event={"ID":"059a38f7-1f72-418d-9c8a-fa3e86a8674c","Type":"ContainerDied","Data":"b57f11d3e247d00e3054d3c698b29b4b6731bf2640f339e680da253bd129bc9b"} Nov 01 22:21:56.727355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.727336 4395 scope.go:122] "RemoveContainer" containerID="26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275" Nov 01 22:21:56.739066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.739011 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv" Nov 01 22:21:56.798258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.798224 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:21:56.799045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:56.798995 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Nov 01 22:21:57.022946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.022918 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 22:21:57.023329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.023098 4395 scope.go:122] "RemoveContainer" containerID="654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795" Nov 01 22:21:57.028199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.028137 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-l89vv"] Nov 01 22:21:57.050664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.050635 4395 scope.go:122] "RemoveContainer" containerID="26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275" Nov 01 22:21:57.051136 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:21:57.051105 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275\": container with ID starting with 26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275 not found: ID does not exist" containerID="26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275" Nov 01 22:21:57.051279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.051147 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275"} err="failed to get container status \"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275\": rpc error: code = NotFound desc = could not find container \"26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275\": container with ID starting with 26340d7297d08dc2504f7c561d04f6a9341427ba3e42b617199403cc886d2275 not found: ID does not exist" Nov 01 22:21:57.051279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.051172 4395 scope.go:122] "RemoveContainer" containerID="654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795" Nov 01 22:21:57.051470 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:21:57.051435 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795\": container with ID starting with 654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795 not found: ID does not exist" containerID="654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795" Nov 01 22:21:57.051529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.051471 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795"} err="failed to get container status \"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795\": rpc error: code = NotFound desc = could not find container \"654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795\": container with ID starting with 654aae9ff2bd9c440b79048d3278846dd71494bdc83936d4c4fc04c875953795 not found: ID does not exist" Nov 01 22:21:57.170385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.170335 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" path="/var/lib/kubelet/pods/059a38f7-1f72-418d-9c8a-fa3e86a8674c/volumes" Nov 01 22:21:57.785326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.785283 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:57.785825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.785804 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:21:57.786734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:21:57.786709 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:22:04.701365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.701329 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6947-305d27"] Nov 01 22:22:04.702210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702190 4395 cpu_manager.go:414] "containerMap: removing container" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:04.702272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702213 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:04.702272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702251 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" Nov 01 22:22:04.702272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702260 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" Nov 01 22:22:04.702457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702442 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39994c2-9b0f-4784-a9c6-16af42387eb3" containerName="discovery" Nov 01 22:22:04.702494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702479 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:04.702536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.702495 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:04.708259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.708238 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:04.712135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.712110 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jm4jc\"" Nov 01 22:22:04.712743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.712719 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6947-305d27"] Nov 01 22:22:04.757791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.757755 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zxr\" (UniqueName: \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\") pod \"test-curl-6947-305d27\" (UID: \"06af2db4-3ce0-4c54-b176-4a6688b46bf5\") " pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:04.859483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.859457 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zxr\" (UniqueName: \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\") pod \"test-curl-6947-305d27\" (UID: \"06af2db4-3ce0-4c54-b176-4a6688b46bf5\") " pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:04.871399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:04.871375 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zxr\" (UniqueName: \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\") pod \"test-curl-6947-305d27\" (UID: \"06af2db4-3ce0-4c54-b176-4a6688b46bf5\") " pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:05.023720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:05.023695 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:05.307191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:05.307162 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6947-305d27"] Nov 01 22:22:05.307346 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:22:05.307255 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06af2db4_3ce0_4c54_b176_4a6688b46bf5.slice/crio-9fbad14c271765c12d50dcba0cc4073852b47db22b3c11df8c30f8ac67af739f WatchSource:0}: Error finding container 9fbad14c271765c12d50dcba0cc4073852b47db22b3c11df8c30f8ac67af739f: Status 404 returned error can't find the container with id 9fbad14c271765c12d50dcba0cc4073852b47db22b3c11df8c30f8ac67af739f Nov 01 22:22:05.309425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:05.309403 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:22:05.879050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:05.878770 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6947-305d27" event={"ID":"06af2db4-3ce0-4c54-b176-4a6688b46bf5","Type":"ContainerStarted","Data":"9fbad14c271765c12d50dcba0cc4073852b47db22b3c11df8c30f8ac67af739f"} Nov 01 22:22:09.910807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:09.910769 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6947-305d27" event={"ID":"06af2db4-3ce0-4c54-b176-4a6688b46bf5","Type":"ContainerStarted","Data":"79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353"} Nov 01 22:22:09.931345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:09.931297 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6947-305d27" podStartSLOduration=2.312154905 podStartE2EDuration="5.931282939s" totalImagesPullingTime="3.619128034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:09.929101342 +0000 UTC m=+2276.903003880" watchObservedRunningTime="2026-11-01 22:22:09.931282939 +0000 UTC m=+2276.905185477" Nov 01 22:22:10.615860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:10.615202 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6947-305d27"] Nov 01 22:22:10.619954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:10.619929 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6947-305d27"] Nov 01 22:22:11.966242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:11.966193 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6947-305d27" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" containerName="test-curl-6947-305d27" containerID="cri-o://79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353" gracePeriod=2 Nov 01 22:22:14.252087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:14.252018 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:14.355270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:14.355241 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\" (UniqueName: \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\") pod \"06af2db4-3ce0-4c54-b176-4a6688b46bf5\" (UID: \"06af2db4-3ce0-4c54-b176-4a6688b46bf5\") " Nov 01 22:22:14.357601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:14.357575 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr" pod "06af2db4-3ce0-4c54-b176-4a6688b46bf5" (UID: "06af2db4-3ce0-4c54-b176-4a6688b46bf5"). InnerVolumeSpecName "kube-api-access-m5zxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:22:14.457668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:14.457640 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zxr\" (UniqueName: \"kubernetes.io/projected/06af2db4-3ce0-4c54-b176-4a6688b46bf5-kube-api-access-m5zxr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:22:15.027670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.027639 4395 generic.go:356] "Generic (PLEG): container finished" podID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" containerID="79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353" exitCode=137 Nov 01 22:22:15.027863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.027684 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:15.027863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.027726 4395 scope.go:122] "RemoveContainer" containerID="79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353" Nov 01 22:22:15.037914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.037898 4395 scope.go:122] "RemoveContainer" containerID="79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353" Nov 01 22:22:15.038235 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:22:15.038214 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353\": container with ID starting with 79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353 not found: ID does not exist" containerID="79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353" Nov 01 22:22:15.038326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.038247 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353"} err="failed to get container status \"79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353\": rpc error: code = NotFound desc = could not find container \"79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353\": container with ID starting with 79b957064915bed5ea54cfddd0e27082b4bb4d7993bec4d2f47b8fe1a5c91353 not found: ID does not exist" Nov 01 22:22:15.038841 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:22:15.038819 4395 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6947-305d27\" is forbidden: User \"system:node:ip-10-0-136-47.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-47.us-east-2.compute.internal' and this object" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" pod="opendatahub/test-curl-6947-305d27" Nov 01 22:22:15.167124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:15.167095 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" path="/var/lib/kubelet/pods/06af2db4-3ce0-4c54-b176-4a6688b46bf5/volumes" Nov 01 22:22:22.302912 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:22:22.302878 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:22:23.121714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.121675 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:22:23.122876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.122853 4395 cpu_manager.go:414] "containerMap: removing container" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:23.122947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.122879 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059a38f7-1f72-418d-9c8a-fa3e86a8674c" containerName="istio-proxy" Nov 01 22:22:23.122947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.122944 4395 cpu_manager.go:414] "containerMap: removing container" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" containerName="test-curl-6947-305d27" Nov 01 22:22:23.123039 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.122953 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" containerName="test-curl-6947-305d27" Nov 01 22:22:23.123372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.123311 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="06af2db4-3ce0-4c54-b176-4a6688b46bf5" containerName="test-curl-6947-305d27" Nov 01 22:22:23.128669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.128649 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.141882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.141860 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6b4c49-openshift-default-dockercfg-9prbz\"" Nov 01 22:22:23.142552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.142530 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:22:23.154942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.154906 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.154991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155019 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgv7\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155087 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155154 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155312 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155344 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155440 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.155563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.155538 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256346 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256461 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256486 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256593 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256631 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256687 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256710 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgv7\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.256921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.256898 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.257084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.257058 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.257153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.257124 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.257311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.257286 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.257432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.257321 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.257432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.257403 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.259098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.259079 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.259197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.259144 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.272290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.272271 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.272702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.272674 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgv7\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\") pod \"e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.445146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.445072 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:23.741063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.740999 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:22:23.741787 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:22:23.741760 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6730a8_5f0c_4f32_8f30_31ee61270b03.slice/crio-18fe0909bc6014bac55d7db96d0f51b7df8e43be945f43e3bd19014328a392bf WatchSource:0}: Error finding container 18fe0909bc6014bac55d7db96d0f51b7df8e43be945f43e3bd19014328a392bf: Status 404 returned error can't find the container with id 18fe0909bc6014bac55d7db96d0f51b7df8e43be945f43e3bd19014328a392bf Nov 01 22:22:23.745001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.744964 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:23.745122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.745067 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:23.745189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:23.745117 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:24.138094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.138015 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" event={"ID":"ee6730a8-5f0c-4f32-8f30-31ee61270b03","Type":"ContainerStarted","Data":"97b843ba0aa6c1d0508f70c6ee1a2c30219f24258bf016ef903eced6e917c0b6"} Nov 01 22:22:24.138094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.138102 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" event={"ID":"ee6730a8-5f0c-4f32-8f30-31ee61270b03","Type":"ContainerStarted","Data":"18fe0909bc6014bac55d7db96d0f51b7df8e43be945f43e3bd19014328a392bf"} Nov 01 22:22:24.165641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.165571 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podStartSLOduration=1.165556451 podStartE2EDuration="1.165556451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:24.161477214 +0000 UTC m=+2291.135379746" watchObservedRunningTime="2026-11-01 22:22:24.165556451 +0000 UTC m=+2291.139458993" Nov 01 22:22:24.452544 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.452423 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:24.453085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.452595 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:24.453085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:24.452642 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:25.455495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:25.455454 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:25.456210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:25.455523 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:26.453056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:26.449423 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:26.453056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:26.449498 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:27.445596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:27.445552 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:27.446160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:27.445627 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:28.445735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:28.445694 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:28.446414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:28.445757 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:29.453065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:29.452490 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:29.453065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:29.452571 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:30.457054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:30.452480 4395 patch_prober.go:29] interesting pod/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 22:22:30.457054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:30.452555 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 22:22:31.244984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.244940 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:22:31.251426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.251391 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.263554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.263525 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5a355b-openshift-default-dockercfg-mfzwg\"" Nov 01 22:22:31.263895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.263866 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421244 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421288 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421318 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421366 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7ws\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421505 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421733 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421769 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.424054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.421874 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.533049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.532907 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546366 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546637 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546769 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546824 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546856 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.546916 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7ws\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.547116 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.547205 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.547235 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.547341 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.548046 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.548496 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.549047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.548890 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.551320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.550158 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.559090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.557052 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.566426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.561399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7ws\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.567832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.567778 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\") pod \"e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.579360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.579327 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:31.629053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.623347 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:31.629053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.625009 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:31.633245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.630725 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:22:31.935135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.934967 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:22:31.938522 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:22:31.938493 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9511b9_6ef3_4b00_9d5a_cfadc532745d.slice/crio-5cece073f5303dcbbe7a3a42b2989cc9828249b2ed8fbab79e01a44590dcb779 WatchSource:0}: Error finding container 5cece073f5303dcbbe7a3a42b2989cc9828249b2ed8fbab79e01a44590dcb779: Status 404 returned error can't find the container with id 5cece073f5303dcbbe7a3a42b2989cc9828249b2ed8fbab79e01a44590dcb779 Nov 01 22:22:31.941815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.941776 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:31.941952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.941851 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:31.941952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:31.941882 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:22:32.307295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:32.307254 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerStarted","Data":"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb"} Nov 01 22:22:32.307295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:32.307320 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerStarted","Data":"5cece073f5303dcbbe7a3a42b2989cc9828249b2ed8fbab79e01a44590dcb779"} Nov 01 22:22:32.335324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:32.335260 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podStartSLOduration=1.335240575 podStartE2EDuration="1.335240575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:32.333499868 +0000 UTC m=+2299.307402409" watchObservedRunningTime="2026-11-01 22:22:32.335240575 +0000 UTC m=+2299.309143114" Nov 01 22:22:32.580585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:32.580507 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:33.587786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:33.587734 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:34.325593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:34.325559 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:34.326515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:34.326496 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:22:35.846700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.846669 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:22:35.852318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.852295 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:35.859752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.859714 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a355b-dockercfg-jk4pd\"" Nov 01 22:22:35.860437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.860392 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a355b\"" Nov 01 22:22:35.862334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.862308 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:22:35.870761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.870738 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:22:35.870883 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.870828 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:35.890644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:35.890600 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:22:36.012093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.010631 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.012093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.010870 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.012093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.010910 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh98c\" (UniqueName: \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.012093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.010944 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rfr\" (UniqueName: \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.119108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.115363 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.119108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.115575 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.119108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.115608 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh98c\" (UniqueName: \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.119108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.115641 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rfr\" (UniqueName: \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.119108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.117072 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.138566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.138503 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.186060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.178382 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rfr\" (UniqueName: \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.219056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.201321 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:36.219056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.206749 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh98c\" (UniqueName: \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\") pod \"payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.475536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.475502 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:36.601683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.601631 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:22:36.609880 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:22:36.609843 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf26761_81ff_43c9_9cba_de82f81b24f7.slice/crio-b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c WatchSource:0}: Error finding container b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c: Status 404 returned error can't find the container with id b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c Nov 01 22:22:36.766365 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:22:36.766333 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb54559_09a3_4eb2_beed_015996442b03.slice/crio-8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d WatchSource:0}: Error finding container 8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d: Status 404 returned error can't find the container with id 8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d Nov 01 22:22:36.766365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:36.766281 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:22:37.355827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.355786 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" event={"ID":"bbf26761-81ff-43c9-9cba-de82f81b24f7","Type":"ContainerStarted","Data":"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2"} Nov 01 22:22:37.356373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.355843 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" event={"ID":"bbf26761-81ff-43c9-9cba-de82f81b24f7","Type":"ContainerStarted","Data":"b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c"} Nov 01 22:22:37.356373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.355879 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:37.357676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.357645 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" event={"ID":"efb54559-09a3-4eb2-beed-015996442b03","Type":"ContainerStarted","Data":"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171"} Nov 01 22:22:37.357832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.357705 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:37.357832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.357729 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" event={"ID":"efb54559-09a3-4eb2-beed-015996442b03","Type":"ContainerStarted","Data":"8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d"} Nov 01 22:22:37.379987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.379937 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" podStartSLOduration=2.379923328 podStartE2EDuration="2.379923328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:37.375499364 +0000 UTC m=+2304.349401927" watchObservedRunningTime="2026-11-01 22:22:37.379923328 +0000 UTC m=+2304.353825867" Nov 01 22:22:37.398201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:37.398150 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" podStartSLOduration=2.398136533 podStartE2EDuration="2.398136533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:37.395586043 +0000 UTC m=+2304.369488583" watchObservedRunningTime="2026-11-01 22:22:37.398136533 +0000 UTC m=+2304.372039088" Nov 01 22:22:48.369196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:48.369165 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:22:48.369727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:48.369501 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:22:52.354123 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:22:52.354086 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:22:53.897783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.890905 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:22:53.914002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.913962 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:53.920128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.920097 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"default-dockercfg-xzpjt\"" Nov 01 22:22:53.920300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.920275 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"sub-test-model-1783e3-kserve-self-signed-certs\"" Nov 01 22:22:53.920392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.920377 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"kube-root-ca.crt\"" Nov 01 22:22:53.920483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.920453 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1783e3\"/\"openshift-service-ca.crt\"" Nov 01 22:22:53.925906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:53.925876 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:22:54.024458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.024458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024457 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.024714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024493 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.024714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024708 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.024825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024727 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6hc\" (UniqueName: \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.024825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.024769 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136572 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6hc\" (UniqueName: \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136881 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136910 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.136961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.137561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.137489 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.150785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.148770 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.150785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.149707 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.183050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.174843 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6hc\" (UniqueName: \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.183050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.177719 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.205050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.204782 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\") pod \"sub-test-model-1783e3-kserve-6f657b876c-fqfvd\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.232499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.232458 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:22:54.646202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:54.646173 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:22:55.694051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:55.683083 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" event={"ID":"12f52ec8-8c56-4bff-9557-33535307f6f4","Type":"ContainerStarted","Data":"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267"} Nov 01 22:22:55.694051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:55.683149 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" event={"ID":"12f52ec8-8c56-4bff-9557-33535307f6f4","Type":"ContainerStarted","Data":"a99cd94a32e8ea5eb22c5bfa0929f3dcccb66339554e09124e488a81532a92d2"} Nov 01 22:22:55.724049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:55.720679 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" podStartSLOduration=2.720659247 podStartE2EDuration="2.720659247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:55.716481836 +0000 UTC m=+2322.690384375" watchObservedRunningTime="2026-11-01 22:22:55.720659247 +0000 UTC m=+2322.694561787" Nov 01 22:22:56.246106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:22:56.241096 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:23:04.236799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:04.236761 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:23:04.249503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:04.249477 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:23:04.832007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:04.831971 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:23:15.149353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:15.148195 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:23:15.157426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:15.157392 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:23:15.420940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:15.420860 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:23:15.421107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:15.421080 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" containerID="cri-o://c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1" gracePeriod=30 Nov 01 22:23:16.418973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:16.418890 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:23:16.432292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:16.432261 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:23:18.756059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.754504 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:23:18.781308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.781275 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:23:18.781945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.781568 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.789792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.789768 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"default-dockercfg-rxnds\"" Nov 01 22:23:18.795793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.791606 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"sub-test-model-e2bf63-kserve-self-signed-certs\"" Nov 01 22:23:18.816486 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.816453 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"openshift-service-ca.crt\"" Nov 01 22:23:18.816664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.816595 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-e2bf63\"/\"kube-root-ca.crt\"" Nov 01 22:23:18.872172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872142 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k945\" (UniqueName: \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.872327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872181 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.872396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872370 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.872578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872561 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.872629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872591 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.872676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.872641 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974231 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974287 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974348 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974507 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k945\" (UniqueName: \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974538 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.974660 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.975251 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.975388 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:18.984052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:18.976172 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:19.004064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:19.001317 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:19.004064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:19.002665 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:19.064113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:19.053653 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k945\" (UniqueName: \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\") pod \"sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:19.128646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:19.128610 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:19.420401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:19.420369 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:23:19.421572 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:23:19.421548 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbe82684_41ae_4d94_ad7c_8c646fc5b271.slice/crio-69a780100e718430d70d8948809b9a8bdbe6fa89497e1946ccc4c4ce8e75dcba WatchSource:0}: Error finding container 69a780100e718430d70d8948809b9a8bdbe6fa89497e1946ccc4c4ce8e75dcba: Status 404 returned error can't find the container with id 69a780100e718430d70d8948809b9a8bdbe6fa89497e1946ccc4c4ce8e75dcba Nov 01 22:23:20.084847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.084814 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" event={"ID":"dbe82684-41ae-4d94-ad7c-8c646fc5b271","Type":"ContainerStarted","Data":"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4"} Nov 01 22:23:20.085344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.084857 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" event={"ID":"dbe82684-41ae-4d94-ad7c-8c646fc5b271","Type":"ContainerStarted","Data":"69a780100e718430d70d8948809b9a8bdbe6fa89497e1946ccc4c4ce8e75dcba"} Nov 01 22:23:20.104638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.104588 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" podStartSLOduration=2.104570633 podStartE2EDuration="2.104570633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:20.103722624 +0000 UTC m=+2347.077625161" watchObservedRunningTime="2026-11-01 22:23:20.104570633 +0000 UTC m=+2347.078473171" Nov 01 22:23:20.128847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.128817 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:20.148777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.148728 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" podUID="efb54559-09a3-4eb2-beed-015996442b03" containerName="payload-processing" containerID="cri-o://0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171" gracePeriod=25 Nov 01 22:23:20.157847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.157821 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" podUID="bbf26761-81ff-43c9-9cba-de82f81b24f7" containerName="payload-pre-processing" containerID="cri-o://e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2" gracePeriod=25 Nov 01 22:23:20.539855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.539828 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:23:20.544507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.544483 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:23:20.611137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611103 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\") pod \"efb54559-09a3-4eb2-beed-015996442b03\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " Nov 01 22:23:20.611343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611325 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\" (UniqueName: \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\") pod \"efb54559-09a3-4eb2-beed-015996442b03\" (UID: \"efb54559-09a3-4eb2-beed-015996442b03\") " Nov 01 22:23:20.611439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611395 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\" (UniqueName: \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\") pod \"bbf26761-81ff-43c9-9cba-de82f81b24f7\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " Nov 01 22:23:20.611584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611477 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\") pod \"bbf26761-81ff-43c9-9cba-de82f81b24f7\" (UID: \"bbf26761-81ff-43c9-9cba-de82f81b24f7\") " Nov 01 22:23:20.611959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611930 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume" pod "bbf26761-81ff-43c9-9cba-de82f81b24f7" (UID: "bbf26761-81ff-43c9-9cba-de82f81b24f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:20.612084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.611554 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume" pod "efb54559-09a3-4eb2-beed-015996442b03" (UID: "efb54559-09a3-4eb2-beed-015996442b03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:20.612448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.612419 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf26761-81ff-43c9-9cba-de82f81b24f7-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.612534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.612449 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efb54559-09a3-4eb2-beed-015996442b03-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.614292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.614268 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c" pod "efb54559-09a3-4eb2-beed-015996442b03" (UID: "efb54559-09a3-4eb2-beed-015996442b03"). InnerVolumeSpecName "kube-api-access-xh98c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:20.614390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.614369 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr" pod "bbf26761-81ff-43c9-9cba-de82f81b24f7" (UID: "bbf26761-81ff-43c9-9cba-de82f81b24f7"). InnerVolumeSpecName "kube-api-access-82rfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:20.714111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.713990 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh98c\" (UniqueName: \"kubernetes.io/projected/efb54559-09a3-4eb2-beed-015996442b03-kube-api-access-xh98c\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.714111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.714020 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rfr\" (UniqueName: \"kubernetes.io/projected/bbf26761-81ff-43c9-9cba-de82f81b24f7-kube-api-access-82rfr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.730850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.730818 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:23:20.815080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815041 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815131 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815299 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815357 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815421 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815451 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:20.815594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815464 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815580 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815680 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815583 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:20.815776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815693 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:20.815776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815734 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\") pod \"b12ec669-a26f-4b71-aeda-e00818e9421b\" (UID: \"b12ec669-a26f-4b71-aeda-e00818e9421b\") " Nov 01 22:23:20.815952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.815807 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:20.816128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816098 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:20.816755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816728 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.816878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816761 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.816878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816776 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.816878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816791 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12ec669-a26f-4b71-aeda-e00818e9421b-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.816878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.816806 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.819055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.818967 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "kube-api-access-w55bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:20.819283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.819257 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:20.819405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.819299 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:20.819923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.819902 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo" pod "b12ec669-a26f-4b71-aeda-e00818e9421b" (UID: "b12ec669-a26f-4b71-aeda-e00818e9421b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:23:20.918081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.918014 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.918081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.918073 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.918081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.918088 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:20.918351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:20.918102 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55bl\" (UniqueName: \"kubernetes.io/projected/b12ec669-a26f-4b71-aeda-e00818e9421b-kube-api-access-w55bl\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:21.149610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.149573 4395 generic.go:356] "Generic (PLEG): container finished" podID="bbf26761-81ff-43c9-9cba-de82f81b24f7" containerID="e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2" exitCode=0 Nov 01 22:23:21.156717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.149673 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" Nov 01 22:23:21.157638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.149695 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" event={"ID":"bbf26761-81ff-43c9-9cba-de82f81b24f7","Type":"ContainerDied","Data":"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2"} Nov 01 22:23:21.157910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.157885 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg" event={"ID":"bbf26761-81ff-43c9-9cba-de82f81b24f7","Type":"ContainerDied","Data":"b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c"} Nov 01 22:23:21.158069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.158050 4395 scope.go:122] "RemoveContainer" containerID="e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2" Nov 01 22:23:21.174324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.174295 4395 generic.go:356] "Generic (PLEG): container finished" podID="efb54559-09a3-4eb2-beed-015996442b03" containerID="0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171" exitCode=0 Nov 01 22:23:21.174454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.174427 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" Nov 01 22:23:21.179723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.179693 4395 generic.go:356] "Generic (PLEG): container finished" podID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerID="c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1" exitCode=0 Nov 01 22:23:21.180709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.180687 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" Nov 01 22:23:21.189813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.189789 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" event={"ID":"efb54559-09a3-4eb2-beed-015996442b03","Type":"ContainerDied","Data":"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171"} Nov 01 22:23:21.189897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.189819 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw" event={"ID":"efb54559-09a3-4eb2-beed-015996442b03","Type":"ContainerDied","Data":"8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d"} Nov 01 22:23:21.189897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.189830 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" event={"ID":"b12ec669-a26f-4b71-aeda-e00818e9421b","Type":"ContainerDied","Data":"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1"} Nov 01 22:23:21.189897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.189853 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4" event={"ID":"b12ec669-a26f-4b71-aeda-e00818e9421b","Type":"ContainerDied","Data":"f8074d1544e5fa3129b715d0fe1c1eba959181a701167ad521d193b28ee7f97f"} Nov 01 22:23:21.191282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.191261 4395 scope.go:122] "RemoveContainer" containerID="e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2" Nov 01 22:23:21.191610 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:21.191579 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2\": container with ID starting with e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2 not found: ID does not exist" containerID="e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2" Nov 01 22:23:21.191701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.191620 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2"} err="failed to get container status \"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2\": rpc error: code = NotFound desc = could not find container \"e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2\": container with ID starting with e564cc9453ee161b21b953c4efcb1c2de1f194b56a490e4a771f8e3ad37c2fa2 not found: ID does not exist" Nov 01 22:23:21.191701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.191642 4395 scope.go:122] "RemoveContainer" containerID="0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171" Nov 01 22:23:21.209182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.209084 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:23:21.209941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.209921 4395 scope.go:122] "RemoveContainer" containerID="0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171" Nov 01 22:23:21.210502 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:21.210480 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171\": container with ID starting with 0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171 not found: ID does not exist" containerID="0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171" Nov 01 22:23:21.210606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.210510 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171"} err="failed to get container status \"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171\": rpc error: code = NotFound desc = could not find container \"0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171\": container with ID starting with 0921cbbfce62197e13a6d321be6c8c2a569a2d9d60b77f648e05d3f089a4b171 not found: ID does not exist" Nov 01 22:23:21.210606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.210532 4395 scope.go:122] "RemoveContainer" containerID="c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1" Nov 01 22:23:21.214451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.214429 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-67b6d8b7f4-j4skw"] Nov 01 22:23:21.224666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.224641 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:23:21.225844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.225674 4395 scope.go:122] "RemoveContainer" containerID="c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1" Nov 01 22:23:21.226124 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:21.226094 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1\": container with ID starting with c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1 not found: ID does not exist" containerID="c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1" Nov 01 22:23:21.226237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.226132 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1"} err="failed to get container status \"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1\": rpc error: code = NotFound desc = could not find container \"c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1\": container with ID starting with c59e9353c20bc67ad8a535aea9fe3ab6cdb35311adc0eda1bab11476b15caad1 not found: ID does not exist" Nov 01 22:23:21.229055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.229003 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-546989879f-jjvgg"] Nov 01 22:23:21.255628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.255586 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:23:21.266001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.265938 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-szlf4"] Nov 01 22:23:21.282610 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:21.282558 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb54559_09a3_4eb2_beed_015996442b03.slice/crio-8809a3d725d1f842a4f64eca315cf4fa59a6ef06ad865dd5c1e746057d612c1d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf26761_81ff_43c9_9cba_de82f81b24f7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf26761_81ff_43c9_9cba_de82f81b24f7.slice/crio-b801ce5958c350298638bcb7382ab442be2c87d7f5add3fd8642c4cbbd50ea5c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12ec669_a26f_4b71_aeda_e00818e9421b.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:23:21.420175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.420131 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" containerID="cri-o://5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4" gracePeriod=25 Nov 01 22:23:21.432875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:21.432840 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" containerID="cri-o://d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37" gracePeriod=25 Nov 01 22:23:22.305149 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:22.305110 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:23:23.167090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:23.167056 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" path="/var/lib/kubelet/pods/b12ec669-a26f-4b71-aeda-e00818e9421b/volumes" Nov 01 22:23:23.167566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:23.167550 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf26761-81ff-43c9-9cba-de82f81b24f7" path="/var/lib/kubelet/pods/bbf26761-81ff-43c9-9cba-de82f81b24f7/volumes" Nov 01 22:23:23.167939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:23.167926 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb54559-09a3-4eb2-beed-015996442b03" path="/var/lib/kubelet/pods/efb54559-09a3-4eb2-beed-015996442b03/volumes" Nov 01 22:23:26.016077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.005762 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:23:26.055066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.055004 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" podUID="ad8bc2bc-648f-48d3-a826-498915ea3add" containerName="maas-api" containerID="cri-o://4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe" gracePeriod=30 Nov 01 22:23:26.816724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.816698 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:23:26.890325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.890230 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\") pod \"ad8bc2bc-648f-48d3-a826-498915ea3add\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " Nov 01 22:23:26.890325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.890284 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\" (UniqueName: \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\") pod \"ad8bc2bc-648f-48d3-a826-498915ea3add\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " Nov 01 22:23:26.890562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.890371 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\") pod \"ad8bc2bc-648f-48d3-a826-498915ea3add\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " Nov 01 22:23:26.890959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.890931 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\") pod \"ad8bc2bc-648f-48d3-a826-498915ea3add\" (UID: \"ad8bc2bc-648f-48d3-a826-498915ea3add\") " Nov 01 22:23:26.891394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.891311 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca" pod "ad8bc2bc-648f-48d3-a826-498915ea3add" (UID: "ad8bc2bc-648f-48d3-a826-498915ea3add"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:26.892726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.892682 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bc2bc-648f-48d3-a826-498915ea3add-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:26.893372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.893347 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd" pod "ad8bc2bc-648f-48d3-a826-498915ea3add" (UID: "ad8bc2bc-648f-48d3-a826-498915ea3add"). InnerVolumeSpecName "kube-api-access-mjdxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:26.893568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.893545 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs" pod "ad8bc2bc-648f-48d3-a826-498915ea3add" (UID: "ad8bc2bc-648f-48d3-a826-498915ea3add"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:26.894817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.894790 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls" pod "ad8bc2bc-648f-48d3-a826-498915ea3add" (UID: "ad8bc2bc-648f-48d3-a826-498915ea3add"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:26.999052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.995293 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:26.999052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.995341 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjdxd\" (UniqueName: \"kubernetes.io/projected/ad8bc2bc-648f-48d3-a826-498915ea3add-kube-api-access-mjdxd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:26.999052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:26.995359 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8bc2bc-648f-48d3-a826-498915ea3add-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:27.582882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.582845 4395 generic.go:356] "Generic (PLEG): container finished" podID="ad8bc2bc-648f-48d3-a826-498915ea3add" containerID="4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe" exitCode=0 Nov 01 22:23:27.583592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.583564 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" event={"ID":"ad8bc2bc-648f-48d3-a826-498915ea3add","Type":"ContainerDied","Data":"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe"} Nov 01 22:23:27.583753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.583738 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" event={"ID":"ad8bc2bc-648f-48d3-a826-498915ea3add","Type":"ContainerDied","Data":"4ddd15946f3d3de2eeb4a56d6a4c1261d1c108a9c0349de5752cbbefb66a501b"} Nov 01 22:23:27.583849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.583838 4395 scope.go:122] "RemoveContainer" containerID="4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe" Nov 01 22:23:27.605256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.605221 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82" Nov 01 22:23:27.646627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.646598 4395 scope.go:122] "RemoveContainer" containerID="4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe" Nov 01 22:23:27.647496 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:27.647052 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe\": container with ID starting with 4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe not found: ID does not exist" containerID="4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe" Nov 01 22:23:27.647496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.647091 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe"} err="failed to get container status \"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe\": rpc error: code = NotFound desc = could not find container \"4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe\": container with ID starting with 4888fa1b4da493ff6d14795a0888cb2770bfc7714331072471e1f887d94993fe not found: ID does not exist" Nov 01 22:23:27.655157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.655123 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:23:27.659921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:27.659893 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7e2df-77b79f87cc-fph82"] Nov 01 22:23:29.131353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:29.131321 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:29.227365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:29.227330 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8bc2bc-648f-48d3-a826-498915ea3add" path="/var/lib/kubelet/pods/ad8bc2bc-648f-48d3-a826-498915ea3add/volumes" Nov 01 22:23:29.228171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:29.228150 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:29.645568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:29.645531 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:23:30.067113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:30.067073 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:23:30.067327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:30.067073 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:23:32.540310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.540275 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:23:32.541052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541019 4395 cpu_manager.go:414] "containerMap: removing container" podUID="efb54559-09a3-4eb2-beed-015996442b03" containerName="payload-processing" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541053 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb54559-09a3-4eb2-beed-015996442b03" containerName="payload-processing" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541067 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541072 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541083 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ad8bc2bc-648f-48d3-a826-498915ea3add" containerName="maas-api" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541089 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8bc2bc-648f-48d3-a826-498915ea3add" containerName="maas-api" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541103 4395 cpu_manager.go:414] "containerMap: removing container" podUID="bbf26761-81ff-43c9-9cba-de82f81b24f7" containerName="payload-pre-processing" Nov 01 22:23:32.541110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541108 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf26761-81ff-43c9-9cba-de82f81b24f7" containerName="payload-pre-processing" Nov 01 22:23:32.541337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541233 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb54559-09a3-4eb2-beed-015996442b03" containerName="payload-processing" Nov 01 22:23:32.541337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541241 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12ec669-a26f-4b71-aeda-e00818e9421b" containerName="istio-proxy" Nov 01 22:23:32.541337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541251 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8bc2bc-648f-48d3-a826-498915ea3add" containerName="maas-api" Nov 01 22:23:32.541337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.541261 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf26761-81ff-43c9-9cba-de82f81b24f7" containerName="payload-pre-processing" Nov 01 22:23:32.546842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.546819 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.553135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.553112 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a355b\"" Nov 01 22:23:32.558882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.558793 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:23:32.566520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.566489 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:23:32.574903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.574848 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.582506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.581867 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a355b-dockercfg-tpmqj\"" Nov 01 22:23:32.582961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.582940 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:23:32.614620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.614591 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmgk\" (UniqueName: \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.614787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.614626 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2rs\" (UniqueName: \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.614868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.614823 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.615023 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.615001 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.736063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.720184 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.736063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.720353 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmgk\" (UniqueName: \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.736063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.720384 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2rs\" (UniqueName: \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.736063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.720601 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.737685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.737652 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:32.770054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.752782 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.869107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.849009 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmgk\" (UniqueName: \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\") pod \"payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:32.918139 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:32.918104 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2rs\" (UniqueName: \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\") pod \"payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:33.166488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:33.166407 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:33.204062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:33.202329 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:33.814546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:33.814509 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:23:33.816624 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:23:33.816595 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6a49b0_570d_47ce_8e4a_df8a5986a248.slice/crio-c309cac664d3a84a5d3763d879cc139952984eb2b10ecd8a2c980e999d181bba WatchSource:0}: Error finding container c309cac664d3a84a5d3763d879cc139952984eb2b10ecd8a2c980e999d181bba: Status 404 returned error can't find the container with id c309cac664d3a84a5d3763d879cc139952984eb2b10ecd8a2c980e999d181bba Nov 01 22:23:33.820524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:33.820474 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:23:34.674002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:34.673960 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" event={"ID":"071c0930-3c76-44d3-8492-7804ec29b763","Type":"ContainerStarted","Data":"edc7b4e46afd97fb2dbfa442d4f44f7e2140693db804ad7f382cbe8d116364be"} Nov 01 22:23:34.703359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:34.703315 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" event={"ID":"fd6a49b0-570d-47ce-8e4a-df8a5986a248","Type":"ContainerStarted","Data":"c309cac664d3a84a5d3763d879cc139952984eb2b10ecd8a2c980e999d181bba"} Nov 01 22:23:36.754965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.754923 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" event={"ID":"fd6a49b0-570d-47ce-8e4a-df8a5986a248","Type":"ContainerStarted","Data":"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb"} Nov 01 22:23:36.755481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.755007 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:36.757416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.757375 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" event={"ID":"071c0930-3c76-44d3-8492-7804ec29b763","Type":"ContainerStarted","Data":"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b"} Nov 01 22:23:36.757582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.757451 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:36.825555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.824570 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" podStartSLOduration=2.531960245 podStartE2EDuration="4.824552923s" totalImagesPullingTime="2.292592678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:36.823428428 +0000 UTC m=+2363.797330968" watchObservedRunningTime="2026-11-01 22:23:36.824552923 +0000 UTC m=+2363.798455463" Nov 01 22:23:36.827813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:36.827763 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" podStartSLOduration=2.525898745 podStartE2EDuration="4.827749435s" totalImagesPullingTime="2.30185069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:36.80534051 +0000 UTC m=+2363.779243051" watchObservedRunningTime="2026-11-01 22:23:36.827749435 +0000 UTC m=+2363.801651975" Nov 01 22:23:40.067608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:40.066443 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:23:40.068102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:40.067608 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:23:41.785731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.785637 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:23:41.796516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.795217 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:23:41.808827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.808566 4395 generic.go:356] "Generic (PLEG): container finished" podID="97e71ccc-afcb-47b8-b973-da12247405fb" containerID="d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37" exitCode=0 Nov 01 22:23:41.808827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.808613 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" Nov 01 22:23:41.808827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.808724 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" event={"ID":"97e71ccc-afcb-47b8-b973-da12247405fb","Type":"ContainerDied","Data":"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37"} Nov 01 22:23:41.808827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.808764 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72" event={"ID":"97e71ccc-afcb-47b8-b973-da12247405fb","Type":"ContainerDied","Data":"b029abcec1e017c742a2b82f4836972be8731e09631678a76503a6fb6afc4fc3"} Nov 01 22:23:41.808827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.808788 4395 scope.go:122] "RemoveContainer" containerID="d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37" Nov 01 22:23:41.815726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.812687 4395 generic.go:356] "Generic (PLEG): container finished" podID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerID="5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4" exitCode=0 Nov 01 22:23:41.815726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.812743 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" event={"ID":"0ed9619c-ae76-4c69-b97f-43a01a181cee","Type":"ContainerDied","Data":"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4"} Nov 01 22:23:41.815726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.812788 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" event={"ID":"0ed9619c-ae76-4c69-b97f-43a01a181cee","Type":"ContainerDied","Data":"fcb276afe11016ed110bc854f1f8f4603a03463283f1c273acec1fa341db5907"} Nov 01 22:23:41.815726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.812865 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6" Nov 01 22:23:41.846481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.846190 4395 scope.go:122] "RemoveContainer" containerID="d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37" Nov 01 22:23:41.848151 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:41.846767 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37\": container with ID starting with d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37 not found: ID does not exist" containerID="d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37" Nov 01 22:23:41.848151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.846801 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37"} err="failed to get container status \"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37\": rpc error: code = NotFound desc = could not find container \"d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37\": container with ID starting with d0ac8092aaa6b40eb03bc98e1bcfe188254fd4688497aa3af90db85a0c30de37 not found: ID does not exist" Nov 01 22:23:41.848151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.846824 4395 scope.go:122] "RemoveContainer" containerID="5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4" Nov 01 22:23:41.863368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.863345 4395 scope.go:122] "RemoveContainer" containerID="5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4" Nov 01 22:23:41.863734 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:41.863698 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4\": container with ID starting with 5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4 not found: ID does not exist" containerID="5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4" Nov 01 22:23:41.863818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.863737 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4"} err="failed to get container status \"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4\": rpc error: code = NotFound desc = could not find container \"5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4\": container with ID starting with 5a7fc32cc42970cd6799cf1e2144b2d57c906d95fe64cbe69b62fb18b39911b4 not found: ID does not exist" Nov 01 22:23:41.866263 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866240 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\") pod \"97e71ccc-afcb-47b8-b973-da12247405fb\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " Nov 01 22:23:41.866409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866389 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\") pod \"0ed9619c-ae76-4c69-b97f-43a01a181cee\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " Nov 01 22:23:41.866479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866464 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\" (UniqueName: \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\") pod \"0ed9619c-ae76-4c69-b97f-43a01a181cee\" (UID: \"0ed9619c-ae76-4c69-b97f-43a01a181cee\") " Nov 01 22:23:41.866750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866722 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume" pod "97e71ccc-afcb-47b8-b973-da12247405fb" (UID: "97e71ccc-afcb-47b8-b973-da12247405fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:41.866820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866771 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\" (UniqueName: \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\") pod \"97e71ccc-afcb-47b8-b973-da12247405fb\" (UID: \"97e71ccc-afcb-47b8-b973-da12247405fb\") " Nov 01 22:23:41.866870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.866815 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume" pod "0ed9619c-ae76-4c69-b97f-43a01a181cee" (UID: "0ed9619c-ae76-4c69-b97f-43a01a181cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:41.867590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.867560 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed9619c-ae76-4c69-b97f-43a01a181cee-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:41.867667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.867597 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97e71ccc-afcb-47b8-b973-da12247405fb-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:41.869013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.868987 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx" pod "0ed9619c-ae76-4c69-b97f-43a01a181cee" (UID: "0ed9619c-ae76-4c69-b97f-43a01a181cee"). InnerVolumeSpecName "kube-api-access-dfmrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:41.869445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.869423 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr" pod "97e71ccc-afcb-47b8-b973-da12247405fb" (UID: "97e71ccc-afcb-47b8-b973-da12247405fb"). InnerVolumeSpecName "kube-api-access-fthvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:41.969726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.969695 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmrx\" (UniqueName: \"kubernetes.io/projected/0ed9619c-ae76-4c69-b97f-43a01a181cee-kube-api-access-dfmrx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:41.969726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:41.969728 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthvr\" (UniqueName: \"kubernetes.io/projected/97e71ccc-afcb-47b8-b973-da12247405fb-kube-api-access-fthvr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:42.143898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:42.143868 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:23:42.159058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:42.159013 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7e2df-69cf56d76c-5lf72"] Nov 01 22:23:42.170259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:42.170234 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:23:42.176328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:42.176303 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7e2df-558bccd986-zktt6"] Nov 01 22:23:43.167144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.167114 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" path="/var/lib/kubelet/pods/0ed9619c-ae76-4c69-b97f-43a01a181cee/volumes" Nov 01 22:23:43.167605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.167590 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" path="/var/lib/kubelet/pods/97e71ccc-afcb-47b8-b973-da12247405fb/volumes" Nov 01 22:23:43.832859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.832784 4395 generic.go:356] "Generic (PLEG): container finished" podID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerID="2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb" exitCode=0 Nov 01 22:23:43.833081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.832854 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerDied","Data":"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb"} Nov 01 22:23:43.833336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.833321 4395 scope.go:122] "RemoveContainer" containerID="2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb" Nov 01 22:23:43.835152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.835127 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:23:43.835227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.835195 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:23:43.835267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:43.835226 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:23:44.847216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:44.847175 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerStarted","Data":"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350"} Nov 01 22:23:45.585638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:45.584683 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:23:45.586703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:45.586366 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:45.586703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:45.586421 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:46.592113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:46.592078 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:46.592717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:46.592686 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:47.585091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:47.585057 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:47.585365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:47.585336 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:47.768402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:47.768374 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:23:47.768789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:47.768440 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:23:48.599478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:48.599135 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:48.599478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:48.599198 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:49.326212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:49.326181 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:23:49.584300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:49.584209 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:49.584619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:49.584590 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:50.594053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:50.593406 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:23:50.594053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:50.593474 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:23:51.585395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:51.585361 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:23:51.586571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:51.586552 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:23:52.303807 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:52.303772 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:23:53.071192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.071161 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:23:53.071932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.071918 4395 cpu_manager.go:414] "containerMap: removing container" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" Nov 01 22:23:53.071987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.071933 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" Nov 01 22:23:53.071987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.071951 4395 cpu_manager.go:414] "containerMap: removing container" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" Nov 01 22:23:53.071987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.071956 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" Nov 01 22:23:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.072102 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed9619c-ae76-4c69-b97f-43a01a181cee" containerName="payload-processing" Nov 01 22:23:53.072146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.072114 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e71ccc-afcb-47b8-b973-da12247405fb" containerName="payload-pre-processing" Nov 01 22:23:53.077717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.077698 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.086173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.086148 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e25317\"" Nov 01 22:23:53.086569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.086511 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e25317-dockercfg-ftwmj\"" Nov 01 22:23:53.087252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.087227 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:23:53.122506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.122471 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.122691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.122670 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79hj\" (UniqueName: \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.235155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.235109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79hj\" (UniqueName: \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.238414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.238384 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.255823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.254957 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.273768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.273735 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79hj\" (UniqueName: \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\") pod \"payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.417105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.410840 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:53.828903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.828871 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:23:53.829875 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:23:53.829847 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7497356_bf15_45e6_b532_bab322220593.slice/crio-0deb9a8cd0a6414d26838c4b5b49727f504da577dcf7554024f1b1c32d0164bc WatchSource:0}: Error finding container 0deb9a8cd0a6414d26838c4b5b49727f504da577dcf7554024f1b1c32d0164bc: Status 404 returned error can't find the container with id 0deb9a8cd0a6414d26838c4b5b49727f504da577dcf7554024f1b1c32d0164bc Nov 01 22:23:53.949527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:53.949482 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" event={"ID":"f7497356-bf15-45e6-b532-bab322220593","Type":"ContainerStarted","Data":"0deb9a8cd0a6414d26838c4b5b49727f504da577dcf7554024f1b1c32d0164bc"} Nov 01 22:23:54.959149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:54.959065 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" event={"ID":"f7497356-bf15-45e6-b532-bab322220593","Type":"ContainerStarted","Data":"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959"} Nov 01 22:23:54.959607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:54.959281 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:23:54.981665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:54.981618 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" podStartSLOduration=1.981604814 podStartE2EDuration="1.981604814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:54.977121478 +0000 UTC m=+2381.951024018" watchObservedRunningTime="2026-11-01 22:23:54.981604814 +0000 UTC m=+2381.955507365" Nov 01 22:23:55.184820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.184782 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:23:55.185102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.184998 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" podUID="4e278919-5a21-46b3-a236-1c9fd18fb395" containerName="maas-api" containerID="cri-o://2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028" gracePeriod=30 Nov 01 22:23:55.813003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.812980 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:23:55.861630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.861602 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\") pod \"4e278919-5a21-46b3-a236-1c9fd18fb395\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " Nov 01 22:23:55.861797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.861711 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\" (UniqueName: \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\") pod \"4e278919-5a21-46b3-a236-1c9fd18fb395\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " Nov 01 22:23:55.861797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.861744 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\") pod \"4e278919-5a21-46b3-a236-1c9fd18fb395\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " Nov 01 22:23:55.861973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.861957 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\") pod \"4e278919-5a21-46b3-a236-1c9fd18fb395\" (UID: \"4e278919-5a21-46b3-a236-1c9fd18fb395\") " Nov 01 22:23:55.862220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.862190 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca" pod "4e278919-5a21-46b3-a236-1c9fd18fb395" (UID: "4e278919-5a21-46b3-a236-1c9fd18fb395"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:55.862749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.862719 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e278919-5a21-46b3-a236-1c9fd18fb395-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:55.864352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.864323 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls" pod "4e278919-5a21-46b3-a236-1c9fd18fb395" (UID: "4e278919-5a21-46b3-a236-1c9fd18fb395"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:55.864425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.864402 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs" pod "4e278919-5a21-46b3-a236-1c9fd18fb395" (UID: "4e278919-5a21-46b3-a236-1c9fd18fb395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:55.864494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.864480 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56" pod "4e278919-5a21-46b3-a236-1c9fd18fb395" (UID: "4e278919-5a21-46b3-a236-1c9fd18fb395"). InnerVolumeSpecName "kube-api-access-4pr56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:55.964432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.964399 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:55.964432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.964436 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pr56\" (UniqueName: \"kubernetes.io/projected/4e278919-5a21-46b3-a236-1c9fd18fb395-kube-api-access-4pr56\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:55.964852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.964451 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e278919-5a21-46b3-a236-1c9fd18fb395-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:55.968954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.968913 4395 generic.go:356] "Generic (PLEG): container finished" podID="4e278919-5a21-46b3-a236-1c9fd18fb395" containerID="2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028" exitCode=0 Nov 01 22:23:55.969134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.968975 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" Nov 01 22:23:55.969134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.968996 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" event={"ID":"4e278919-5a21-46b3-a236-1c9fd18fb395","Type":"ContainerDied","Data":"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028"} Nov 01 22:23:55.969134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.969087 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5" event={"ID":"4e278919-5a21-46b3-a236-1c9fd18fb395","Type":"ContainerDied","Data":"0342f34c45a9ce0bdfd37b9f1244fb42a1a6f8d90b1014ef82ee6c1c099386c0"} Nov 01 22:23:55.969134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.969119 4395 scope.go:122] "RemoveContainer" containerID="2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028" Nov 01 22:23:55.980446 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.980428 4395 scope.go:122] "RemoveContainer" containerID="2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028" Nov 01 22:23:55.980712 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:23:55.980695 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028\": container with ID starting with 2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028 not found: ID does not exist" containerID="2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028" Nov 01 22:23:55.980763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.980718 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028"} err="failed to get container status \"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028\": rpc error: code = NotFound desc = could not find container \"2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028\": container with ID starting with 2dd21ff181a44bb8d7f400ec6bcb013b48ca0863f58856786b05d59d48b78028 not found: ID does not exist" Nov 01 22:23:55.998938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:55.998912 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:23:56.003121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:56.003100 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6b4c49-8589d5dd97-fwdz5"] Nov 01 22:23:57.168192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:57.168163 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e278919-5a21-46b3-a236-1c9fd18fb395" path="/var/lib/kubelet/pods/4e278919-5a21-46b3-a236-1c9fd18fb395/volumes" Nov 01 22:23:58.119624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.119593 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:23:58.120420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.120404 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4e278919-5a21-46b3-a236-1c9fd18fb395" containerName="maas-api" Nov 01 22:23:58.120478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.120422 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e278919-5a21-46b3-a236-1c9fd18fb395" containerName="maas-api" Nov 01 22:23:58.120549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.120540 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e278919-5a21-46b3-a236-1c9fd18fb395" containerName="maas-api" Nov 01 22:23:58.126074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.126055 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.134314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.134288 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad047b\"" Nov 01 22:23:58.134612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.134586 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad047b-dockercfg-v9mhh\"" Nov 01 22:23:58.136903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.136878 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:23:58.154875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.154841 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:23:58.160734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.160714 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.169720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.169693 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:23:58.247065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.234193 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.247065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.234348 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqtf\" (UniqueName: \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.247065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.234394 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99cx\" (UniqueName: \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.247065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.234533 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.356053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.339164 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.356053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.339310 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqtf\" (UniqueName: \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.356053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.339356 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99cx\" (UniqueName: \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.356053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.339494 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.356053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.340484 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.375421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.361335 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.395651 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.393830 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqtf\" (UniqueName: \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.423816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.423771 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99cx\" (UniqueName: \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\") pod \"payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.456053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.454311 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:58.506718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.505480 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:58.801762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.801734 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:23:58.802489 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:23:58.802464 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef31f67_45d7_4466_9791_03e45d041c12.slice/crio-0c08b7676f966b881f9ca9b1c530eeffb47c511c132287079482a4ba8279d73b WatchSource:0}: Error finding container 0c08b7676f966b881f9ca9b1c530eeffb47c511c132287079482a4ba8279d73b: Status 404 returned error can't find the container with id 0c08b7676f966b881f9ca9b1c530eeffb47c511c132287079482a4ba8279d73b Nov 01 22:23:58.818382 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:23:58.818341 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467217ac_a35b_489e_808f_0af77225233f.slice/crio-82075f745b053586289b121338c85bdb6221eac2d1d1311f6218e8ab52694eae WatchSource:0}: Error finding container 82075f745b053586289b121338c85bdb6221eac2d1d1311f6218e8ab52694eae: Status 404 returned error can't find the container with id 82075f745b053586289b121338c85bdb6221eac2d1d1311f6218e8ab52694eae Nov 01 22:23:58.819432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:58.819408 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:23:59.012943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.012908 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" event={"ID":"467217ac-a35b-489e-808f-0af77225233f","Type":"ContainerStarted","Data":"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629"} Nov 01 22:23:59.013161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.012968 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" event={"ID":"467217ac-a35b-489e-808f-0af77225233f","Type":"ContainerStarted","Data":"82075f745b053586289b121338c85bdb6221eac2d1d1311f6218e8ab52694eae"} Nov 01 22:23:59.013161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.013000 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:23:59.014666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.014635 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" event={"ID":"def31f67-45d7-4466-9791-03e45d041c12","Type":"ContainerStarted","Data":"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f"} Nov 01 22:23:59.014787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.014688 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" event={"ID":"def31f67-45d7-4466-9791-03e45d041c12","Type":"ContainerStarted","Data":"0c08b7676f966b881f9ca9b1c530eeffb47c511c132287079482a4ba8279d73b"} Nov 01 22:23:59.014787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.014725 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:23:59.059383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.059341 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" podStartSLOduration=1.05932834 podStartE2EDuration="1.05932834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:59.056214567 +0000 UTC m=+2386.030117106" watchObservedRunningTime="2026-11-01 22:23:59.05932834 +0000 UTC m=+2386.033230880" Nov 01 22:23:59.119055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:23:59.106672 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" podStartSLOduration=1.106650747 podStartE2EDuration="1.106650747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:59.102655285 +0000 UTC m=+2386.076557826" watchObservedRunningTime="2026-11-01 22:23:59.106650747 +0000 UTC m=+2386.080553287" Nov 01 22:24:03.061609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:03.061574 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:24:03.062113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:03.061857 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" containerID="cri-o://97b843ba0aa6c1d0508f70c6ee1a2c30219f24258bf016ef903eced6e917c0b6" gracePeriod=30 Nov 01 22:24:05.971975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:05.971945 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:24:08.207497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.207442 4395 generic.go:356] "Generic (PLEG): container finished" podID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerID="97b843ba0aa6c1d0508f70c6ee1a2c30219f24258bf016ef903eced6e917c0b6" exitCode=0 Nov 01 22:24:08.207862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.207522 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" event={"ID":"ee6730a8-5f0c-4f32-8f30-31ee61270b03","Type":"ContainerDied","Data":"97b843ba0aa6c1d0508f70c6ee1a2c30219f24258bf016ef903eced6e917c0b6"} Nov 01 22:24:08.374172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.374147 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:24:08.501839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.501714 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.501839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.501827 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.501913 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.501966 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.502006 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.502236 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.502284 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.502344 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.502459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.502401 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\") pod \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\" (UID: \"ee6730a8-5f0c-4f32-8f30-31ee61270b03\") " Nov 01 22:24:08.508234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.507991 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.508416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.508292 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.508535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.508514 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.524504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.524468 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.524853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.524820 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:08.536101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.526313 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:08.536101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.528829 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:24:08.541226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.540758 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "kube-api-access-tcgv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:08.541226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.541193 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token" pod "ee6730a8-5f0c-4f32-8f30-31ee61270b03" (UID: "ee6730a8-5f0c-4f32-8f30-31ee61270b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604300 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604327 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604338 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604348 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604358 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604366 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604374 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgv7\" (UniqueName: \"kubernetes.io/projected/ee6730a8-5f0c-4f32-8f30-31ee61270b03-kube-api-access-tcgv7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604385 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:08.604658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:08.604393 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6730a8-5f0c-4f32-8f30-31ee61270b03-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:09.217562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:09.217534 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" Nov 01 22:24:09.218063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:09.217566 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw" event={"ID":"ee6730a8-5f0c-4f32-8f30-31ee61270b03","Type":"ContainerDied","Data":"18fe0909bc6014bac55d7db96d0f51b7df8e43be945f43e3bd19014328a392bf"} Nov 01 22:24:09.218063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:09.217639 4395 scope.go:122] "RemoveContainer" containerID="97b843ba0aa6c1d0508f70c6ee1a2c30219f24258bf016ef903eced6e917c0b6" Nov 01 22:24:09.249045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:09.249000 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:24:09.254243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:09.254210 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6b4c49-openshift-default-656f9585df-w6zfw"] Nov 01 22:24:10.076804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.076775 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:24:10.095613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.095581 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:24:10.364317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.364232 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:24:10.365039 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.365012 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" Nov 01 22:24:10.365097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.365042 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" Nov 01 22:24:10.365175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.365153 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" containerName="istio-proxy" Nov 01 22:24:10.370004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.369984 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.377064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.377005 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-09656e\"" Nov 01 22:24:10.377254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.377240 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-09656e-dockercfg-ttkql\"" Nov 01 22:24:10.381569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.381545 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:24:10.413347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.413318 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:24:10.419176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.419157 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.427166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.427120 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:24:10.519154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.509200 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.519154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.509292 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvc6\" (UniqueName: \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.519154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.509360 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.519154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.509753 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrkht\" (UniqueName: \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.613180 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkht\" (UniqueName: \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.613285 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.613341 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvc6\" (UniqueName: \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.613393 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.614475 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.619099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.615567 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.719680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.719644 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkht\" (UniqueName: \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\") pod \"payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.721383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.721342 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:10.736179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.735961 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvc6\" (UniqueName: \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\") pod \"payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:10.743796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:10.743704 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:11.031513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.031481 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:24:11.031893 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:11.031863 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd939f47_c7f7_43f5_9605_1d699405b433.slice/crio-04b90b874692796e744bbc09937b37172b397465e119a8ae723820bf8829f425 WatchSource:0}: Error finding container 04b90b874692796e744bbc09937b37172b397465e119a8ae723820bf8829f425: Status 404 returned error can't find the container with id 04b90b874692796e744bbc09937b37172b397465e119a8ae723820bf8829f425 Nov 01 22:24:11.051725 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:11.051698 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9252994_2494_4839_b1c2_887c887ff7ef.slice/crio-704c7c5b18219af8d838291a1f14e9a35ed6e1eb94ec95865292ab1f6a65d82d WatchSource:0}: Error finding container 704c7c5b18219af8d838291a1f14e9a35ed6e1eb94ec95865292ab1f6a65d82d: Status 404 returned error can't find the container with id 704c7c5b18219af8d838291a1f14e9a35ed6e1eb94ec95865292ab1f6a65d82d Nov 01 22:24:11.052445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.052278 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:24:11.237402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.237368 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6730a8-5f0c-4f32-8f30-31ee61270b03" path="/var/lib/kubelet/pods/ee6730a8-5f0c-4f32-8f30-31ee61270b03/volumes" Nov 01 22:24:11.240077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.240050 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" event={"ID":"fd939f47-c7f7-43f5-9605-1d699405b433","Type":"ContainerStarted","Data":"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403"} Nov 01 22:24:11.240220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.240107 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" event={"ID":"fd939f47-c7f7-43f5-9605-1d699405b433","Type":"ContainerStarted","Data":"04b90b874692796e744bbc09937b37172b397465e119a8ae723820bf8829f425"} Nov 01 22:24:11.240327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.240308 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:11.241887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.241867 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" event={"ID":"f9252994-2494-4839-b1c2-887c887ff7ef","Type":"ContainerStarted","Data":"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058"} Nov 01 22:24:11.242006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.241907 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" event={"ID":"f9252994-2494-4839-b1c2-887c887ff7ef","Type":"ContainerStarted","Data":"704c7c5b18219af8d838291a1f14e9a35ed6e1eb94ec95865292ab1f6a65d82d"} Nov 01 22:24:11.242006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.241935 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:11.261514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.261471 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" podStartSLOduration=1.261457224 podStartE2EDuration="1.261457224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:11.257339305 +0000 UTC m=+2398.231241845" watchObservedRunningTime="2026-11-01 22:24:11.261457224 +0000 UTC m=+2398.235359763" Nov 01 22:24:11.281398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:11.281117 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" podStartSLOduration=1.281106324 podStartE2EDuration="1.281106324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:11.275561538 +0000 UTC m=+2398.249464077" watchObservedRunningTime="2026-11-01 22:24:11.281106324 +0000 UTC m=+2398.255008863" Nov 01 22:24:15.844142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.844110 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:24:15.850393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.850368 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:15.859879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.859852 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:24:15.864075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.861749 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ad047b-serving-cert\"" Nov 01 22:24:15.995042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.994996 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:15.995220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.995084 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:15.995279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.995227 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:15.995279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:15.995251 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5m4\" (UniqueName: \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.097046 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.096949 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.097204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.097111 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.097204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.097143 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5m4\" (UniqueName: \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.097393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.097378 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.097647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.097626 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.099853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.099832 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.099948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.099897 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.111369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.111345 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5m4\" (UniqueName: \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\") pod \"maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.146775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.146749 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:24:16.155437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.155414 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.168105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.163134 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-829cb0-dockercfg-56q77\"" Nov 01 22:24:16.168105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.163285 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-829cb0\"" Nov 01 22:24:16.168105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.164142 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:24:16.206052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.182882 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:16.483097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.481152 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88f9\" (UniqueName: \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.483097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.481329 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.585220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.585176 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88f9\" (UniqueName: \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.585772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.585377 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.586442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.586414 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.603915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.603764 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88f9\" (UniqueName: \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\") pod \"payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.693016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.692939 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:16.864197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:16.856424 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:24:16.910565 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:16.910528 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fe3b52_ca4e_453f_a309_682e457da54a.slice/crio-4c7f7a71affcd4296627a623e74fbe504077db35b2b2bbe89e95ce2911da7cb0 WatchSource:0}: Error finding container 4c7f7a71affcd4296627a623e74fbe504077db35b2b2bbe89e95ce2911da7cb0: Status 404 returned error can't find the container with id 4c7f7a71affcd4296627a623e74fbe504077db35b2b2bbe89e95ce2911da7cb0 Nov 01 22:24:17.021630 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:17.021599 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc39af5a_9db3_4bde_b63b_1bc2f60754d8.slice/crio-26c64283d54e853d6a64e3d40f499b24f82d8498e27d1e0eaf2a7f74cc27f05d WatchSource:0}: Error finding container 26c64283d54e853d6a64e3d40f499b24f82d8498e27d1e0eaf2a7f74cc27f05d: Status 404 returned error can't find the container with id 26c64283d54e853d6a64e3d40f499b24f82d8498e27d1e0eaf2a7f74cc27f05d Nov 01 22:24:17.021957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.021933 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:24:17.488231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.488198 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" event={"ID":"cc39af5a-9db3-4bde-b63b-1bc2f60754d8","Type":"ContainerStarted","Data":"a01f04e363b82a27dfebb226b75308094171f45fa080059d0adab6483286d97a"} Nov 01 22:24:17.488342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.488262 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" event={"ID":"cc39af5a-9db3-4bde-b63b-1bc2f60754d8","Type":"ContainerStarted","Data":"26c64283d54e853d6a64e3d40f499b24f82d8498e27d1e0eaf2a7f74cc27f05d"} Nov 01 22:24:17.488936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.488916 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:17.491287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.491263 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" event={"ID":"f9fe3b52-ca4e-453f-a309-682e457da54a","Type":"ContainerStarted","Data":"4c7f7a71affcd4296627a623e74fbe504077db35b2b2bbe89e95ce2911da7cb0"} Nov 01 22:24:17.512401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:17.512362 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" podStartSLOduration=1.512348571 podStartE2EDuration="1.512348571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:17.507389959 +0000 UTC m=+2404.481292499" watchObservedRunningTime="2026-11-01 22:24:17.512348571 +0000 UTC m=+2404.486251114" Nov 01 22:24:18.500405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:18.500368 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" event={"ID":"f9fe3b52-ca4e-453f-a309-682e457da54a","Type":"ContainerStarted","Data":"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67"} Nov 01 22:24:18.500785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:18.500460 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:18.521787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:18.521725 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" podStartSLOduration=3.007546938 podStartE2EDuration="3.521710207s" totalImagesPullingTime="514.163269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:18.520415635 +0000 UTC m=+2405.494323846" watchObservedRunningTime="2026-11-01 22:24:18.521710207 +0000 UTC m=+2405.495612746" Nov 01 22:24:20.652047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.648267 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:24:20.671049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.661468 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.676531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.674848 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e25317-openshift-default-dockercfg-v8cbj\"" Nov 01 22:24:20.678352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.678327 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:24:20.730772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.730731 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.730923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.730797 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.730923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.730907 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lrf\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731020 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731234 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731283 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731363 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731428 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.731513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.731496 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.832935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.832906 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.832961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833038 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833225 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833269 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833300 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lrf\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833332 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833389 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833456 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833490 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.833731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833676 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.834349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.833868 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.834349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.834212 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.834349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.834287 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.835756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.835734 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.835876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.835859 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.845382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.845360 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lrf\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:20.845382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:20.845374 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:21.010520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.010487 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:21.333392 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:21.333348 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bbe5ce_b472_47f5_a567_7536d5d4ea02.slice/crio-eca64b6394a54391d9cb18cdd6298892c9cc736126651a1cd71149bb38eedf65 WatchSource:0}: Error finding container eca64b6394a54391d9cb18cdd6298892c9cc736126651a1cd71149bb38eedf65: Status 404 returned error can't find the container with id eca64b6394a54391d9cb18cdd6298892c9cc736126651a1cd71149bb38eedf65 Nov 01 22:24:21.335497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.335467 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:24:21.336629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.336595 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:24:21.336738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.336666 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:24:21.336738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.336697 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:24:21.533409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.533372 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" event={"ID":"21bbe5ce-b472-47f5-a567-7536d5d4ea02","Type":"ContainerStarted","Data":"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53"} Nov 01 22:24:21.533578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.533434 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" event={"ID":"21bbe5ce-b472-47f5-a567-7536d5d4ea02","Type":"ContainerStarted","Data":"eca64b6394a54391d9cb18cdd6298892c9cc736126651a1cd71149bb38eedf65"} Nov 01 22:24:21.556950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:21.556894 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podStartSLOduration=1.5568749849999999 podStartE2EDuration="1.556874985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:21.554543105 +0000 UTC m=+2408.528445656" watchObservedRunningTime="2026-11-01 22:24:21.556874985 +0000 UTC m=+2408.530777524" Nov 01 22:24:22.011242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:22.011205 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:22.013155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:22.013130 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:22.013255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:22.013180 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:22.305127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:22.304391 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:24:22.308066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:22.307988 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:24:22.333543 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:24:22.333513 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:24:23.011674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:23.011636 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:23.012112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:23.011703 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:24.011058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:24.010992 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:24.011283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:24.011082 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:24.513134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:24.513096 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:24:25.011778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:25.011748 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:25.011929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:25.011798 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:26.012159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:26.012114 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:26.012577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:26.012179 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:27.014422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:27.014384 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 22:24:27.015138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:27.015106 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 22:24:28.016047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:28.016004 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:28.016496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:28.016408 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:28.017245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:28.017223 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:24:28.504312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:28.504289 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:24:29.656448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.656412 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:24:29.664132 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.664105 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.672737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.672711 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-09656e-serving-cert\"" Nov 01 22:24:29.676843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.676816 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:24:29.779942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.779898 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.780144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.780093 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnmd\" (UniqueName: \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.780311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.780291 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.780377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.780324 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.882155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.882120 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnmd\" (UniqueName: \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.882325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.882307 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.882391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.882346 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.883108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.883083 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.900054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.885896 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.900054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.886107 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.923716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.923607 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:29.947048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:29.937155 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnmd\" (UniqueName: \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\") pod \"maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:30.030124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:30.030082 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:30.333144 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:24:30.333103 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f2108a_798d_4342_a93b_b1af56582e90.slice/crio-85bd54d3327616bc277ba5b4c1f0e345db83887fbc96b474a750b410ef742ba3 WatchSource:0}: Error finding container 85bd54d3327616bc277ba5b4c1f0e345db83887fbc96b474a750b410ef742ba3: Status 404 returned error can't find the container with id 85bd54d3327616bc277ba5b4c1f0e345db83887fbc96b474a750b410ef742ba3 Nov 01 22:24:30.333956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:30.333925 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:24:30.765247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:30.765214 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" event={"ID":"36f2108a-798d-4342-a93b-b1af56582e90","Type":"ContainerStarted","Data":"85bd54d3327616bc277ba5b4c1f0e345db83887fbc96b474a750b410ef742ba3"} Nov 01 22:24:31.919052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:31.903509 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" event={"ID":"36f2108a-798d-4342-a93b-b1af56582e90","Type":"ContainerStarted","Data":"9bb97c9916c671ec1405de43067362d8e27edcccdb1ec02f45031c49f32f430b"} Nov 01 22:24:31.919052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:31.904932 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:31.963097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:31.958434 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" podStartSLOduration=2.538533099 podStartE2EDuration="2.958412199s" totalImagesPullingTime="419.8791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:31.949274836 +0000 UTC m=+2418.923177424" watchObservedRunningTime="2026-11-01 22:24:31.958412199 +0000 UTC m=+2418.932314739" Nov 01 22:24:38.937261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:38.937210 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:24:42.411258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:42.411226 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:24:42.427365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:42.427331 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:24:47.411547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.411511 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" podUID="467217ac-a35b-489e-808f-0af77225233f" containerName="payload-processing" containerID="cri-o://7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629" gracePeriod=25 Nov 01 22:24:47.428071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.428010 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" podUID="def31f67-45d7-4466-9791-03e45d041c12" containerName="payload-pre-processing" containerID="cri-o://769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f" gracePeriod=25 Nov 01 22:24:47.731391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.731366 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:24:47.780695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.780667 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:24:47.842352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842325 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\") pod \"467217ac-a35b-489e-808f-0af77225233f\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " Nov 01 22:24:47.842520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842417 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\") pod \"def31f67-45d7-4466-9791-03e45d041c12\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " Nov 01 22:24:47.842594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842548 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\" (UniqueName: \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\") pod \"def31f67-45d7-4466-9791-03e45d041c12\" (UID: \"def31f67-45d7-4466-9791-03e45d041c12\") " Nov 01 22:24:47.842655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842593 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\" (UniqueName: \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\") pod \"467217ac-a35b-489e-808f-0af77225233f\" (UID: \"467217ac-a35b-489e-808f-0af77225233f\") " Nov 01 22:24:47.842808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842780 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume" pod "467217ac-a35b-489e-808f-0af77225233f" (UID: "467217ac-a35b-489e-808f-0af77225233f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:47.842877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.842829 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume" pod "def31f67-45d7-4466-9791-03e45d041c12" (UID: "def31f67-45d7-4466-9791-03e45d041c12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:47.843324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.843299 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467217ac-a35b-489e-808f-0af77225233f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:47.843444 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.843328 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def31f67-45d7-4466-9791-03e45d041c12-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:47.845175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.845149 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx" pod "467217ac-a35b-489e-808f-0af77225233f" (UID: "467217ac-a35b-489e-808f-0af77225233f"). InnerVolumeSpecName "kube-api-access-l99cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:47.845435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.845416 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf" pod "def31f67-45d7-4466-9791-03e45d041c12" (UID: "def31f67-45d7-4466-9791-03e45d041c12"). InnerVolumeSpecName "kube-api-access-qqqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:47.944993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.944905 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqtf\" (UniqueName: \"kubernetes.io/projected/def31f67-45d7-4466-9791-03e45d041c12-kube-api-access-qqqtf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:47.944993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:47.944937 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99cx\" (UniqueName: \"kubernetes.io/projected/467217ac-a35b-489e-808f-0af77225233f-kube-api-access-l99cx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:48.175587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.175549 4395 generic.go:356] "Generic (PLEG): container finished" podID="467217ac-a35b-489e-808f-0af77225233f" containerID="7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629" exitCode=0 Nov 01 22:24:48.175777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.175593 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" Nov 01 22:24:48.175777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.175624 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" event={"ID":"467217ac-a35b-489e-808f-0af77225233f","Type":"ContainerDied","Data":"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629"} Nov 01 22:24:48.175777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.175672 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq" event={"ID":"467217ac-a35b-489e-808f-0af77225233f","Type":"ContainerDied","Data":"82075f745b053586289b121338c85bdb6221eac2d1d1311f6218e8ab52694eae"} Nov 01 22:24:48.175777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.175694 4395 scope.go:122] "RemoveContainer" containerID="7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629" Nov 01 22:24:48.177293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.177269 4395 generic.go:356] "Generic (PLEG): container finished" podID="def31f67-45d7-4466-9791-03e45d041c12" containerID="769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f" exitCode=0 Nov 01 22:24:48.177418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.177326 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" Nov 01 22:24:48.177418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.177354 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" event={"ID":"def31f67-45d7-4466-9791-03e45d041c12","Type":"ContainerDied","Data":"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f"} Nov 01 22:24:48.177547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.177417 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm" event={"ID":"def31f67-45d7-4466-9791-03e45d041c12","Type":"ContainerDied","Data":"0c08b7676f966b881f9ca9b1c530eeffb47c511c132287079482a4ba8279d73b"} Nov 01 22:24:48.188142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.188125 4395 scope.go:122] "RemoveContainer" containerID="7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629" Nov 01 22:24:48.188463 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:24:48.188439 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629\": container with ID starting with 7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629 not found: ID does not exist" containerID="7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629" Nov 01 22:24:48.188573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.188475 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629"} err="failed to get container status \"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629\": rpc error: code = NotFound desc = could not find container \"7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629\": container with ID starting with 7219409c8906a80653e3bd0f3b915236f328fbcf77a0efc336b7db92ba166629 not found: ID does not exist" Nov 01 22:24:48.188573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.188493 4395 scope.go:122] "RemoveContainer" containerID="769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f" Nov 01 22:24:48.202960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.202937 4395 scope.go:122] "RemoveContainer" containerID="769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f" Nov 01 22:24:48.203553 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:24:48.203522 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f\": container with ID starting with 769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f not found: ID does not exist" containerID="769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f" Nov 01 22:24:48.203708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.203566 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f"} err="failed to get container status \"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f\": rpc error: code = NotFound desc = could not find container \"769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f\": container with ID starting with 769aba3caf10373509163a38014d76837de0d87deaaf74ed789e5075383ad70f not found: ID does not exist" Nov 01 22:24:48.212814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.212787 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:24:48.219370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.219350 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-678ff75974-mh7cq"] Nov 01 22:24:48.224365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.224343 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:24:48.228937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:48.228917 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-54f599749f-5z9zm"] Nov 01 22:24:49.168544 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:49.168513 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467217ac-a35b-489e-808f-0af77225233f" path="/var/lib/kubelet/pods/467217ac-a35b-489e-808f-0af77225233f/volumes" Nov 01 22:24:49.168939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:49.168923 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def31f67-45d7-4466-9791-03e45d041c12" path="/var/lib/kubelet/pods/def31f67-45d7-4466-9791-03e45d041c12/volumes" Nov 01 22:24:52.307615 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:24:52.307579 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:24:54.607164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:54.599915 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:24:54.646280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:54.646217 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:24:59.601122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:59.601083 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" podUID="f9252994-2494-4839-b1c2-887c887ff7ef" containerName="payload-processing" containerID="cri-o://44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058" gracePeriod=25 Nov 01 22:24:59.647302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:59.647264 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" podUID="fd939f47-c7f7-43f5-9605-1d699405b433" containerName="payload-pre-processing" containerID="cri-o://b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403" gracePeriod=25 Nov 01 22:24:59.999291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:24:59.999263 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:25:00.002835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.002799 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:25:00.003655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003637 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f9252994-2494-4839-b1c2-887c887ff7ef" containerName="payload-processing" Nov 01 22:25:00.003742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003657 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9252994-2494-4839-b1c2-887c887ff7ef" containerName="payload-processing" Nov 01 22:25:00.003742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003676 4395 cpu_manager.go:414] "containerMap: removing container" podUID="def31f67-45d7-4466-9791-03e45d041c12" containerName="payload-pre-processing" Nov 01 22:25:00.003742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003681 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def31f67-45d7-4466-9791-03e45d041c12" containerName="payload-pre-processing" Nov 01 22:25:00.003742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003709 4395 cpu_manager.go:414] "containerMap: removing container" podUID="467217ac-a35b-489e-808f-0af77225233f" containerName="payload-processing" Nov 01 22:25:00.003742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003714 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467217ac-a35b-489e-808f-0af77225233f" containerName="payload-processing" Nov 01 22:25:00.003901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003828 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9252994-2494-4839-b1c2-887c887ff7ef" containerName="payload-processing" Nov 01 22:25:00.003901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003838 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="def31f67-45d7-4466-9791-03e45d041c12" containerName="payload-pre-processing" Nov 01 22:25:00.003901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.003849 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="467217ac-a35b-489e-808f-0af77225233f" containerName="payload-processing" Nov 01 22:25:00.007579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.007561 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.022680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.022660 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad047b-dockercfg-v89dc\"" Nov 01 22:25:00.022680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.022658 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad047b\"" Nov 01 22:25:00.023760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.023740 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:25:00.025409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.025337 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:25:00.076542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.076507 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\") pod \"fd939f47-c7f7-43f5-9605-1d699405b433\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " Nov 01 22:25:00.076747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.076619 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\") pod \"f9252994-2494-4839-b1c2-887c887ff7ef\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " Nov 01 22:25:00.076887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.076867 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\" (UniqueName: \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\") pod \"fd939f47-c7f7-43f5-9605-1d699405b433\" (UID: \"fd939f47-c7f7-43f5-9605-1d699405b433\") " Nov 01 22:25:00.076977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.076936 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\" (UniqueName: \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\") pod \"f9252994-2494-4839-b1c2-887c887ff7ef\" (UID: \"f9252994-2494-4839-b1c2-887c887ff7ef\") " Nov 01 22:25:00.077059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.076987 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume" pod "fd939f47-c7f7-43f5-9605-1d699405b433" (UID: "fd939f47-c7f7-43f5-9605-1d699405b433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:00.077132 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.077076 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume" pod "f9252994-2494-4839-b1c2-887c887ff7ef" (UID: "f9252994-2494-4839-b1c2-887c887ff7ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:00.077239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.077214 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.077309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.077272 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjvb\" (UniqueName: \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.077653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.077626 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd939f47-c7f7-43f5-9605-1d699405b433-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:00.077763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.077657 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9252994-2494-4839-b1c2-887c887ff7ef-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:00.081939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.081902 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6" pod "f9252994-2494-4839-b1c2-887c887ff7ef" (UID: "f9252994-2494-4839-b1c2-887c887ff7ef"). InnerVolumeSpecName "kube-api-access-vnvc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:00.083832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.083807 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht" pod "fd939f47-c7f7-43f5-9605-1d699405b433" (UID: "fd939f47-c7f7-43f5-9605-1d699405b433"). InnerVolumeSpecName "kube-api-access-rrkht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:00.187156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.181356 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.272052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.252263 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjvb\" (UniqueName: \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.272052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.252947 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrkht\" (UniqueName: \"kubernetes.io/projected/fd939f47-c7f7-43f5-9605-1d699405b433-kube-api-access-rrkht\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:00.272052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.253733 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.272052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.253773 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvc6\" (UniqueName: \"kubernetes.io/projected/f9252994-2494-4839-b1c2-887c887ff7ef-kube-api-access-vnvc6\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:00.311739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.305326 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjvb\" (UniqueName: \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\") pod \"payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.378092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.377456 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:00.398389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.398350 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:25:00.524232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.524132 4395 generic.go:356] "Generic (PLEG): container finished" podID="fd939f47-c7f7-43f5-9605-1d699405b433" containerID="b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403" exitCode=0 Nov 01 22:25:00.524432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.524340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" event={"ID":"fd939f47-c7f7-43f5-9605-1d699405b433","Type":"ContainerDied","Data":"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403"} Nov 01 22:25:00.524432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.524424 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" event={"ID":"fd939f47-c7f7-43f5-9605-1d699405b433","Type":"ContainerDied","Data":"04b90b874692796e744bbc09937b37172b397465e119a8ae723820bf8829f425"} Nov 01 22:25:00.524547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.524466 4395 scope.go:122] "RemoveContainer" containerID="b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403" Nov 01 22:25:00.524691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.524664 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm" Nov 01 22:25:00.527381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.527333 4395 generic.go:356] "Generic (PLEG): container finished" podID="f9252994-2494-4839-b1c2-887c887ff7ef" containerID="44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058" exitCode=0 Nov 01 22:25:00.527719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.527608 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" event={"ID":"f9252994-2494-4839-b1c2-887c887ff7ef","Type":"ContainerDied","Data":"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058"} Nov 01 22:25:00.527719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.527679 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" event={"ID":"f9252994-2494-4839-b1c2-887c887ff7ef","Type":"ContainerDied","Data":"704c7c5b18219af8d838291a1f14e9a35ed6e1eb94ec95865292ab1f6a65d82d"} Nov 01 22:25:00.527864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.527788 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv" Nov 01 22:25:00.610753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.610718 4395 scope.go:122] "RemoveContainer" containerID="b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403" Nov 01 22:25:00.612195 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:25:00.612155 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403\": container with ID starting with b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403 not found: ID does not exist" containerID="b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403" Nov 01 22:25:00.612313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.612205 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403"} err="failed to get container status \"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403\": rpc error: code = NotFound desc = could not find container \"b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403\": container with ID starting with b35afb1f320e05b5c0ad9de6607a56182860c2eff58fd3dec86acbd17c1a6403 not found: ID does not exist" Nov 01 22:25:00.612313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.612229 4395 scope.go:122] "RemoveContainer" containerID="44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058" Nov 01 22:25:00.620825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.620799 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:25:00.632539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.632511 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-79bd965b6c-q6fdm"] Nov 01 22:25:00.654737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.651557 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:25:00.654737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.652299 4395 scope.go:122] "RemoveContainer" containerID="44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058" Nov 01 22:25:00.654737 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:25:00.652674 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058\": container with ID starting with 44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058 not found: ID does not exist" containerID="44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058" Nov 01 22:25:00.654737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.652701 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058"} err="failed to get container status \"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058\": rpc error: code = NotFound desc = could not find container \"44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058\": container with ID starting with 44d76788c8528f725182664c4e51e848caa9aa59cf1713be461d4096647bb058 not found: ID does not exist" Nov 01 22:25:00.658379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.658350 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-54fcb9c5db-4qctv"] Nov 01 22:25:00.915756 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:00.915720 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aeb3fe_7b12_47f8_a18c_a14f9b2090cf.slice/crio-d79b34960afb1af4ca6504ef30a5e30e148ce8c73a34ff5dcd377e0bf19b2c49 WatchSource:0}: Error finding container d79b34960afb1af4ca6504ef30a5e30e148ce8c73a34ff5dcd377e0bf19b2c49: Status 404 returned error can't find the container with id d79b34960afb1af4ca6504ef30a5e30e148ce8c73a34ff5dcd377e0bf19b2c49 Nov 01 22:25:00.959532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:00.959464 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:25:01.167819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.167744 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9252994-2494-4839-b1c2-887c887ff7ef" path="/var/lib/kubelet/pods/f9252994-2494-4839-b1c2-887c887ff7ef/volumes" Nov 01 22:25:01.168276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.168259 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd939f47-c7f7-43f5-9605-1d699405b433" path="/var/lib/kubelet/pods/fd939f47-c7f7-43f5-9605-1d699405b433/volumes" Nov 01 22:25:01.541327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.541293 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" event={"ID":"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf","Type":"ContainerStarted","Data":"e2a30f81fe7b9b581c0c7791d7242e5a150a539249eefbcc756ff5b4c2b9bc47"} Nov 01 22:25:01.541327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.541342 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" event={"ID":"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf","Type":"ContainerStarted","Data":"d79b34960afb1af4ca6504ef30a5e30e148ce8c73a34ff5dcd377e0bf19b2c49"} Nov 01 22:25:01.541606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.541378 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:01.565836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:01.563322 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" podStartSLOduration=2.563304562 podStartE2EDuration="2.563304562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:01.560621565 +0000 UTC m=+2448.534524105" watchObservedRunningTime="2026-11-01 22:25:01.563304562 +0000 UTC m=+2448.537207103" Nov 01 22:25:05.398753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.398706 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" podUID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" containerName="payload-processing" containerID="cri-o://a01f04e363b82a27dfebb226b75308094171f45fa080059d0adab6483286d97a" gracePeriod=25 Nov 01 22:25:05.578168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.578125 4395 generic.go:356] "Generic (PLEG): container finished" podID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" containerID="a01f04e363b82a27dfebb226b75308094171f45fa080059d0adab6483286d97a" exitCode=0 Nov 01 22:25:05.578283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.578230 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" event={"ID":"cc39af5a-9db3-4bde-b63b-1bc2f60754d8","Type":"ContainerDied","Data":"a01f04e363b82a27dfebb226b75308094171f45fa080059d0adab6483286d97a"} Nov 01 22:25:05.700163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.700140 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:25:05.751187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.751151 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\") pod \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " Nov 01 22:25:05.751368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.751285 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\" (UniqueName: \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\") pod \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\" (UID: \"cc39af5a-9db3-4bde-b63b-1bc2f60754d8\") " Nov 01 22:25:05.754978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.754942 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9" pod "cc39af5a-9db3-4bde-b63b-1bc2f60754d8" (UID: "cc39af5a-9db3-4bde-b63b-1bc2f60754d8"). InnerVolumeSpecName "kube-api-access-l88f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:05.755425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.755401 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume" pod "cc39af5a-9db3-4bde-b63b-1bc2f60754d8" (UID: "cc39af5a-9db3-4bde-b63b-1bc2f60754d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:05.755611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.755588 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:25:05.757265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757241 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" containerName="payload-processing" Nov 01 22:25:05.757387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757268 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" containerName="payload-processing" Nov 01 22:25:05.757387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757289 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fd939f47-c7f7-43f5-9605-1d699405b433" containerName="payload-pre-processing" Nov 01 22:25:05.757387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757297 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd939f47-c7f7-43f5-9605-1d699405b433" containerName="payload-pre-processing" Nov 01 22:25:05.757559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757540 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" containerName="payload-processing" Nov 01 22:25:05.757559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.757556 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd939f47-c7f7-43f5-9605-1d699405b433" containerName="payload-pre-processing" Nov 01 22:25:05.766576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.763069 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.766576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.766257 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:25:05.774399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.774369 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:25:05.796227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.796194 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:25:05.796572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.796547 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.854887 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.854976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855060 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855162 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855193 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk9w\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855274 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj55k\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855303 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855337 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855439 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855498 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855554 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855588 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855672 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855715 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855747 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855796 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855877 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855963 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:05.860092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.855981 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l88f9\" (UniqueName: \"kubernetes.io/projected/cc39af5a-9db3-4bde-b63b-1bc2f60754d8-kube-api-access-l88f9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:05.957963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.957870 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.957963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.957938 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.957971 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk9w\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958066 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj55k\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958093 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958129 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958232 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958290 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958329 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958359 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958417 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958438 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958441 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958509 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958538 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.958853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958565 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.958853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958650 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.958852 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959066 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959100 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.959236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959209 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.959236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959216 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.959373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959279 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959549 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.959578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959573 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.959836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.959816 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.960957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.960932 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.961292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.961272 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.961702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.961683 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.962081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.962060 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.970538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.970517 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.970721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.970701 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk9w\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:05.971087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.971064 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj55k\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:05.971253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:05.971237 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:06.107045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.107001 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:06.115622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.115599 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:06.406580 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:06.406537 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8e6cb3_854f_4431_9e03_4cfcc5f03daa.slice/crio-cd67356d2818bbe9e0c641ebc57e34d4c8675232c88c08b4f16c6961a740b89e WatchSource:0}: Error finding container cd67356d2818bbe9e0c641ebc57e34d4c8675232c88c08b4f16c6961a740b89e: Status 404 returned error can't find the container with id cd67356d2818bbe9e0c641ebc57e34d4c8675232c88c08b4f16c6961a740b89e Nov 01 22:25:06.406948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.406618 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:25:06.410902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.410863 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.411209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.411116 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.411209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.411168 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.414773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.414750 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:25:06.415188 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:06.415164 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6c0a3_4a25_4be0_9f1c_04f5f2ef95e3.slice/crio-3ed1c0fefbeca1df0d64d108a60ea1205bd97aa009dfa0321721c51408e3bfe7 WatchSource:0}: Error finding container 3ed1c0fefbeca1df0d64d108a60ea1205bd97aa009dfa0321721c51408e3bfe7: Status 404 returned error can't find the container with id 3ed1c0fefbeca1df0d64d108a60ea1205bd97aa009dfa0321721c51408e3bfe7 Nov 01 22:25:06.423672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.423620 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.423799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.423719 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.423799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.423762 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:25:06.595538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.595499 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" event={"ID":"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa","Type":"ContainerStarted","Data":"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd"} Nov 01 22:25:06.595745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.595575 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" event={"ID":"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa","Type":"ContainerStarted","Data":"cd67356d2818bbe9e0c641ebc57e34d4c8675232c88c08b4f16c6961a740b89e"} Nov 01 22:25:06.598916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.598883 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" event={"ID":"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3","Type":"ContainerStarted","Data":"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791"} Nov 01 22:25:06.599174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.598942 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" event={"ID":"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3","Type":"ContainerStarted","Data":"3ed1c0fefbeca1df0d64d108a60ea1205bd97aa009dfa0321721c51408e3bfe7"} Nov 01 22:25:06.601539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.601338 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" Nov 01 22:25:06.601539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.601364 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98" event={"ID":"cc39af5a-9db3-4bde-b63b-1bc2f60754d8","Type":"ContainerDied","Data":"26c64283d54e853d6a64e3d40f499b24f82d8498e27d1e0eaf2a7f74cc27f05d"} Nov 01 22:25:06.601539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.601427 4395 scope.go:122] "RemoveContainer" containerID="a01f04e363b82a27dfebb226b75308094171f45fa080059d0adab6483286d97a" Nov 01 22:25:06.623912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.623856 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podStartSLOduration=1.6238376589999999 podStartE2EDuration="1.623837659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:06.622402387 +0000 UTC m=+2453.596304926" watchObservedRunningTime="2026-11-01 22:25:06.623837659 +0000 UTC m=+2453.597740203" Nov 01 22:25:06.648601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.646141 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:25:06.652962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.652881 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6f767dc558-nxb98"] Nov 01 22:25:06.678082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:06.673356 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podStartSLOduration=1.673337274 podStartE2EDuration="1.673337274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:06.666483731 +0000 UTC m=+2453.640386276" watchObservedRunningTime="2026-11-01 22:25:06.673337274 +0000 UTC m=+2453.647239815" Nov 01 22:25:07.107232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.107197 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:07.108543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.108522 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:07.108593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.108578 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:07.115944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.115920 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:07.117134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.117113 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:07.117244 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.117158 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:07.166793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:07.166763 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc39af5a-9db3-4bde-b63b-1bc2f60754d8" path="/var/lib/kubelet/pods/cc39af5a-9db3-4bde-b63b-1bc2f60754d8/volumes" Nov 01 22:25:08.108237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:08.108189 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:08.108754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:08.108287 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:08.116349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:08.116316 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:08.116506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:08.116373 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:09.111052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:09.110912 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:09.111052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:09.110980 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:09.126052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:09.117907 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:09.126052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:09.117972 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:10.107565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:10.107526 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:10.107763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:10.107599 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:10.116149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:10.116119 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:10.116554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:10.116165 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:11.108114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:11.108070 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:11.108315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:11.108151 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:11.116002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:11.115971 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:11.116173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:11.116022 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:12.107355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:12.107321 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:12.107560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:12.107378 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:12.116573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:12.116546 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:12.116976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:12.116597 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:12.552596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:12.552560 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:25:13.108061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.107999 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:13.108270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.108094 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:13.116664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.116626 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:25:13.117139 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.116694 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" Nov 01 22:25:13.817473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.817434 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:25:13.822393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.822366 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:13.829872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.827813 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-09656e\"" Nov 01 22:25:13.829872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.829226 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-09656e-dockercfg-6j5mv\"" Nov 01 22:25:13.833389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.832831 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:25:13.868746 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.868682 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thf9\" (UniqueName: \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:13.868924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.868909 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:13.979732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:13.979689 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thf9\" (UniqueName: \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.007528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.007467 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.009094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.008716 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.091655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.091572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thf9\" (UniqueName: \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\") pod \"payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.107670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.107633 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:14.108280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.108249 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:14.153320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.153284 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.167668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.167631 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:14.170279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.169932 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:14.170925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.170878 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:25:14.533098 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:14.533049 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4330ae_04f2_4189_9fc2_09c72940a5e2.slice/crio-c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453 WatchSource:0}: Error finding container c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453: Status 404 returned error can't find the container with id c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453 Nov 01 22:25:14.535843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.535801 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:25:14.675778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.675741 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" event={"ID":"2a4330ae-04f2-4189-9fc2-09c72940a5e2","Type":"ContainerStarted","Data":"80410bbfa96e9f9668473e974381a707ce4874cdaa6ad3ae361e5f2926d80657"} Nov 01 22:25:14.676012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.675800 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" event={"ID":"2a4330ae-04f2-4189-9fc2-09c72940a5e2","Type":"ContainerStarted","Data":"c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453"} Nov 01 22:25:14.676096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.676057 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:14.698929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:14.698867 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" podStartSLOduration=1.698848866 podStartE2EDuration="1.698848866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:14.693837403 +0000 UTC m=+2461.667739942" watchObservedRunningTime="2026-11-01 22:25:14.698848866 +0000 UTC m=+2461.672751407" Nov 01 22:25:15.108306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:15.108269 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:15.108494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:15.108322 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:16.107366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:16.107332 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:16.107837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:16.107406 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:17.107768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:17.107727 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:17.108192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:17.107793 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:17.576582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:17.576551 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:25:17.576801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:17.576766 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" containerID="cri-o://f801bd48f2a73a39e5f11f3385209744bda6849b8e690bbeaf126c86055ce1e0" gracePeriod=30 Nov 01 22:25:18.107354 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:18.107323 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:18.107530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:18.107373 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:19.107549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:19.107518 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:19.107949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:19.107575 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:20.107470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:20.107436 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:20.107657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:20.107519 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:21.107900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:21.107861 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:21.108397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:21.108191 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:21.108397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:21.108236 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:22.107319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.107289 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:22.107524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.107340 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:22.306997 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:25:22.306961 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:25:22.745370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.745334 4395 generic.go:356] "Generic (PLEG): container finished" podID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerID="f801bd48f2a73a39e5f11f3385209744bda6849b8e690bbeaf126c86055ce1e0" exitCode=0 Nov 01 22:25:22.745525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.745385 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" event={"ID":"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37","Type":"ContainerDied","Data":"f801bd48f2a73a39e5f11f3385209744bda6849b8e690bbeaf126c86055ce1e0"} Nov 01 22:25:22.871926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.871899 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:25:22.973043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.972998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973117 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973191 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973310 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:22.973397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973319 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973368 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973401 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973426 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973660 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.973683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973668 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:22.974102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973706 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\") pod \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\" (UID: \"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37\") " Nov 01 22:25:22.974102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.973885 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:22.974248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974209 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:22.974441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974416 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:22.974854 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974829 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:22.974945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974857 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:22.974945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974873 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:22.974945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974886 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:22.974945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.974900 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:22.975691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.975660 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:22.975782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.975736 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:25:22.975840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.975787 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "kube-api-access-zr795". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:22.976062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:22.976018 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token" pod "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" (UID: "b8f06aaa-bc7e-48bd-871d-9b09cd1efb37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:23.077458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.077426 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr795\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-kube-api-access-zr795\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:23.077458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.077456 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:23.077660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.077468 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:23.077660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.077476 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:23.107770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.107695 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:23.107770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.107758 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:23.754482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.754446 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" Nov 01 22:25:23.754482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.754471 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4" event={"ID":"b8f06aaa-bc7e-48bd-871d-9b09cd1efb37","Type":"ContainerDied","Data":"439efe6b5a50707f4471c5d83f24fc125681b196291b8e4d4529ee01ccc954cf"} Nov 01 22:25:23.754970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.754544 4395 scope.go:122] "RemoveContainer" containerID="f801bd48f2a73a39e5f11f3385209744bda6849b8e690bbeaf126c86055ce1e0" Nov 01 22:25:23.777005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.776977 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:25:23.781167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:23.781144 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-jdxd4"] Nov 01 22:25:24.108542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:24.108450 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:24.108727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:24.108645 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:25.108100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:25.108068 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:25.108483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:25.108116 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:25.167809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:25.167778 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" path="/var/lib/kubelet/pods/b8f06aaa-bc7e-48bd-871d-9b09cd1efb37/volumes" Nov 01 22:25:25.686605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:25.686576 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:25:26.037087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.037055 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:25:26.037858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.037844 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" Nov 01 22:25:26.037907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.037861 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" Nov 01 22:25:26.038008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.037998 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f06aaa-bc7e-48bd-871d-9b09cd1efb37" containerName="istio-proxy" Nov 01 22:25:26.043273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.043253 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.047274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.047250 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-829cb0-dockercfg-ztjkk\"" Nov 01 22:25:26.050136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.050111 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-829cb0\"" Nov 01 22:25:26.051381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.051359 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:25:26.118789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.110424 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:26.118789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.110484 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:26.340293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.340181 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.373062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.348585 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxlp\" (UniqueName: \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.385065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.362923 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:25:26.485134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.485096 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.485671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.485646 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxlp\" (UniqueName: \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.495306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.495275 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.512133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.511177 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:25:26.512133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.511184 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.520154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.519942 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxlp\" (UniqueName: \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.620295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.620200 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.620295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.620251 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vdf\" (UniqueName: \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.664285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.664255 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:26.722443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.722413 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.722762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.722470 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vdf\" (UniqueName: \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.723577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.723517 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.735902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.735872 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vdf\" (UniqueName: \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\") pod \"payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:26.848840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:26.848793 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:27.034714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.034685 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:25:27.108759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.107938 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:27.108759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.107982 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:27.212250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.212207 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:25:27.818049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.817743 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" event={"ID":"b90f0c03-589b-4a47-84d3-257348fa517f","Type":"ContainerStarted","Data":"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5"} Nov 01 22:25:27.818049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.817818 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" event={"ID":"b90f0c03-589b-4a47-84d3-257348fa517f","Type":"ContainerStarted","Data":"ff3d57569ca15a2783e20d5d6d05de9d7b11e5713234e70a527d39c75ad1b379"} Nov 01 22:25:27.818049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.817866 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:27.822365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.819677 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" event={"ID":"9ec612db-5ae2-477e-84bc-71a5239b8a5c","Type":"ContainerStarted","Data":"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371"} Nov 01 22:25:27.822365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.819728 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" event={"ID":"9ec612db-5ae2-477e-84bc-71a5239b8a5c","Type":"ContainerStarted","Data":"b423fa8840e7375960dada274fd5727ad3442452bd385b42214c721c19a7feaa"} Nov 01 22:25:27.822365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.819750 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:27.837790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.837731 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" podStartSLOduration=1.8377154500000001 podStartE2EDuration="1.83771545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:27.836079152 +0000 UTC m=+2474.809981691" watchObservedRunningTime="2026-11-01 22:25:27.83771545 +0000 UTC m=+2474.811617990" Nov 01 22:25:27.855398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:27.855351 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" podStartSLOduration=1.8553361210000001 podStartE2EDuration="1.855336121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:27.854474877 +0000 UTC m=+2474.828377418" watchObservedRunningTime="2026-11-01 22:25:27.855336121 +0000 UTC m=+2474.829238662" Nov 01 22:25:28.108141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:28.107379 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:28.108141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:28.107437 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:29.107514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:29.107473 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:29.108005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:29.107532 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:30.107472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:30.107441 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:30.107636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:30.107492 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:31.107794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:31.107760 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 01 22:25:31.108307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:31.107808 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 01 22:25:32.112222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:32.112194 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:32.113325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:32.113303 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:25:38.145482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:38.145434 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:25:38.146585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:38.146555 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" containerID="cri-o://9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d" gracePeriod=30 Nov 01 22:25:38.832577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:38.832547 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:25:38.833002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:38.832978 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:25:39.176001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:39.175926 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:25:39.176001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:39.175992 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:25:43.856506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.856482 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:25:43.907202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907167 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907246 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907368 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907478 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907536 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907570 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.907705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907643 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907787 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907930 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.907979 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\") pod \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\" (UID: \"39894a71-8d71-4cab-bbcb-2ec5aeceed9f\") " Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.908799 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.908896 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:43.909161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.908980 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:43.909569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.909234 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:43.909853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.909819 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:43.910626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.910587 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:25:43.910732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.910672 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:43.911434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.911410 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:43.911434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.911415 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh" pod "39894a71-8d71-4cab-bbcb-2ec5aeceed9f" (UID: "39894a71-8d71-4cab-bbcb-2ec5aeceed9f"). InnerVolumeSpecName "kube-api-access-xvldh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:43.957572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.957539 4395 generic.go:356] "Generic (PLEG): container finished" podID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerID="9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d" exitCode=0 Nov 01 22:25:43.957722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.957610 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" Nov 01 22:25:43.957722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.957616 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" event={"ID":"39894a71-8d71-4cab-bbcb-2ec5aeceed9f","Type":"ContainerDied","Data":"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d"} Nov 01 22:25:43.957722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.957679 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j" event={"ID":"39894a71-8d71-4cab-bbcb-2ec5aeceed9f","Type":"ContainerDied","Data":"e796f324fd46a81ab3e7615895a6546741cef253bd1e39154bd93d341811f02d"} Nov 01 22:25:43.957722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.957705 4395 scope.go:122] "RemoveContainer" containerID="9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d" Nov 01 22:25:43.969767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.969750 4395 scope.go:122] "RemoveContainer" containerID="9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d" Nov 01 22:25:43.970062 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:25:43.970022 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d\": container with ID starting with 9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d not found: ID does not exist" containerID="9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d" Nov 01 22:25:43.970197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.970075 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d"} err="failed to get container status \"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d\": rpc error: code = NotFound desc = could not find container \"9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d\": container with ID starting with 9f8370db67ed526780bbc404d739bed58e97c5dc010393e440834a6e8219766d not found: ID does not exist" Nov 01 22:25:43.987469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.987446 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:25:43.999785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:43.999752 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-s6f5j"] Nov 01 22:25:44.010934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010891 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.010934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010933 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.010934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010944 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.011205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010954 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.011205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010964 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.011205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010973 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvldh\" (UniqueName: \"kubernetes.io/projected/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-kube-api-access-xvldh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.011205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010983 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:44.011205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:44.010992 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39894a71-8d71-4cab-bbcb-2ec5aeceed9f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:45.168286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:45.168256 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" path="/var/lib/kubelet/pods/39894a71-8d71-4cab-bbcb-2ec5aeceed9f/volumes" Nov 01 22:25:47.003969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.003941 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:25:47.004798 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.004779 4395 cpu_manager.go:414] "containerMap: removing container" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" Nov 01 22:25:47.004856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.004810 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" Nov 01 22:25:47.005090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.005075 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="39894a71-8d71-4cab-bbcb-2ec5aeceed9f" containerName="istio-proxy" Nov 01 22:25:47.010473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.010455 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.018983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.018936 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:25:47.019928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.019902 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0bb375\"" Nov 01 22:25:47.024961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.024936 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:25:47.030824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.030803 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.040095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.040072 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:25:47.040308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.040288 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0bb375-dockercfg-h4m5s\"" Nov 01 22:25:47.076513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.076480 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwdx\" (UniqueName: \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.076693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.076674 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.219815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.219776 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.220013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.219843 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.220164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.220082 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld49\" (UniqueName: \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.220231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.220163 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwdx\" (UniqueName: \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.249242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.249161 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.270326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.260706 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwdx\" (UniqueName: \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.330051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.325884 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.330051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.326334 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld49\" (UniqueName: \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.330051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.329097 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.334406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.331436 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.356678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.356645 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld49\" (UniqueName: \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\") pod \"payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.357963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.357935 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:47.652601 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:47.652572 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda419fbeb_e8d7_4253_8451_ae4ec39cda6a.slice/crio-3cd7bd7b99811264c3705a940bda7fb0a65bf77e8a4fb612c5e7c9dbc695934a WatchSource:0}: Error finding container 3cd7bd7b99811264c3705a940bda7fb0a65bf77e8a4fb612c5e7c9dbc695934a: Status 404 returned error can't find the container with id 3cd7bd7b99811264c3705a940bda7fb0a65bf77e8a4fb612c5e7c9dbc695934a Nov 01 22:25:47.653161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.653138 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:25:47.675838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.675807 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:25:47.679397 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:25:47.679361 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f23c08b_0908_45c9_9b98_4f3cf30dc371.slice/crio-23ddd8b31997d102dd8656159c0db2caa663bfebfa642153c85754c71408b6bc WatchSource:0}: Error finding container 23ddd8b31997d102dd8656159c0db2caa663bfebfa642153c85754c71408b6bc: Status 404 returned error can't find the container with id 23ddd8b31997d102dd8656159c0db2caa663bfebfa642153c85754c71408b6bc Nov 01 22:25:47.996273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.996155 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" event={"ID":"a419fbeb-e8d7-4253-8451-ae4ec39cda6a","Type":"ContainerStarted","Data":"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5"} Nov 01 22:25:47.996273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.996219 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" event={"ID":"a419fbeb-e8d7-4253-8451-ae4ec39cda6a","Type":"ContainerStarted","Data":"3cd7bd7b99811264c3705a940bda7fb0a65bf77e8a4fb612c5e7c9dbc695934a"} Nov 01 22:25:47.996273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.996251 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:25:47.998570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.998545 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" event={"ID":"3f23c08b-0908-45c9-9b98-4f3cf30dc371","Type":"ContainerStarted","Data":"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2"} Nov 01 22:25:47.998685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.998584 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" event={"ID":"3f23c08b-0908-45c9-9b98-4f3cf30dc371","Type":"ContainerStarted","Data":"23ddd8b31997d102dd8656159c0db2caa663bfebfa642153c85754c71408b6bc"} Nov 01 22:25:47.998685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:47.998617 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:48.015615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:48.015558 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" podStartSLOduration=2.015544441 podStartE2EDuration="2.015544441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:48.014483073 +0000 UTC m=+2494.988385625" watchObservedRunningTime="2026-11-01 22:25:48.015544441 +0000 UTC m=+2494.989446980" Nov 01 22:25:48.038809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:48.038769 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" podStartSLOduration=1.038756049 podStartE2EDuration="1.038756049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:48.036239312 +0000 UTC m=+2495.010141868" watchObservedRunningTime="2026-11-01 22:25:48.038756049 +0000 UTC m=+2495.012658606" Nov 01 22:25:52.304725 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:25:52.304691 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:25:59.010245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:59.010208 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:25:59.010735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:25:59.010714 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:26:08.837873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.837842 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:26:08.844621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.844595 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:08.849257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.849238 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a00c6a-dockercfg-hbtw6\"" Nov 01 22:26:08.851110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.851088 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:26:08.855642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.855003 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a00c6a\"" Nov 01 22:26:08.877699 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.877673 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:26:08.885588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.885562 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:26:08.885717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.885668 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:08.920724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.920610 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:08.920724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.920660 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:08.928933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.920789 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkk5k\" (UniqueName: \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:08.930729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:08.930704 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprm7\" (UniqueName: \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.037657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.035522 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.037657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.035576 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:09.037657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.035704 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk5k\" (UniqueName: \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:09.037657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.035977 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprm7\" (UniqueName: \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.056475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.056439 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.057485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.057433 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:09.111670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.111601 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk5k\" (UniqueName: \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\") pod \"payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:09.128276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.127740 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprm7\" (UniqueName: \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.169061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.169019 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:09.207923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.207732 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:09.481563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.481527 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:26:09.481826 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:26:09.481798 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cfab37_10eb_4ac8_9972_45366c8098d0.slice/crio-eb09b9ab41c2171fc112b3ac12ede73423aa4fa87220ccddb9f577dcbdef5330 WatchSource:0}: Error finding container eb09b9ab41c2171fc112b3ac12ede73423aa4fa87220ccddb9f577dcbdef5330: Status 404 returned error can't find the container with id eb09b9ab41c2171fc112b3ac12ede73423aa4fa87220ccddb9f577dcbdef5330 Nov 01 22:26:09.520368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:09.520344 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:26:09.525418 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:26:09.525388 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad9d39c_e5bf_4279_b9dd_0459084442a9.slice/crio-cee2c87b83e71950e58eba0dee6321990eb1c1a7db45ce54ea902162cd9f48b3 WatchSource:0}: Error finding container cee2c87b83e71950e58eba0dee6321990eb1c1a7db45ce54ea902162cd9f48b3: Status 404 returned error can't find the container with id cee2c87b83e71950e58eba0dee6321990eb1c1a7db45ce54ea902162cd9f48b3 Nov 01 22:26:10.245678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.245643 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" event={"ID":"cad9d39c-e5bf-4279-b9dd-0459084442a9","Type":"ContainerStarted","Data":"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7"} Nov 01 22:26:10.246184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.245706 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" event={"ID":"cad9d39c-e5bf-4279-b9dd-0459084442a9","Type":"ContainerStarted","Data":"cee2c87b83e71950e58eba0dee6321990eb1c1a7db45ce54ea902162cd9f48b3"} Nov 01 22:26:10.246184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.245735 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:10.247901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.247874 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" event={"ID":"14cfab37-10eb-4ac8-9972-45366c8098d0","Type":"ContainerStarted","Data":"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f"} Nov 01 22:26:10.248168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.247920 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" event={"ID":"14cfab37-10eb-4ac8-9972-45366c8098d0","Type":"ContainerStarted","Data":"eb09b9ab41c2171fc112b3ac12ede73423aa4fa87220ccddb9f577dcbdef5330"} Nov 01 22:26:10.248168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.247941 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:10.268609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.268565 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" podStartSLOduration=2.268552121 podStartE2EDuration="2.268552121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:10.263931114 +0000 UTC m=+2517.237833688" watchObservedRunningTime="2026-11-01 22:26:10.268552121 +0000 UTC m=+2517.242454660" Nov 01 22:26:10.285944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:10.285898 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" podStartSLOduration=2.285883548 podStartE2EDuration="2.285883548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:10.281438562 +0000 UTC m=+2517.255341102" watchObservedRunningTime="2026-11-01 22:26:10.285883548 +0000 UTC m=+2517.259786086" Nov 01 22:26:21.305802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:21.305774 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:26:21.306264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:21.305835 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:26:21.661582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:21.661502 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:26:21.830897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:21.830870 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:26:21.996274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:21.996208 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg_9ec612db-5ae2-477e-84bc-71a5239b8a5c/payload-processing/0.log" Nov 01 22:26:22.312550 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:22.312512 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:26:33.205912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.205874 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:26:33.212896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.212870 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.224378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.224350 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a68ac-dockercfg-bcwfx\"" Nov 01 22:26:33.227429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.226081 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a68ac\"" Nov 01 22:26:33.231943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.231904 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:26:33.290535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.290503 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.290701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.290681 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshqr\" (UniqueName: \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.415082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.395775 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.415082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.397011 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshqr\" (UniqueName: \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.415082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.396734 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.418770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.418731 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshqr\" (UniqueName: \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:33.543389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:33.543332 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:34.181053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:34.175407 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:26:34.580865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:34.580829 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" event={"ID":"d57d79b3-a752-4808-9558-eef9087cd525","Type":"ContainerStarted","Data":"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289"} Nov 01 22:26:34.581278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:34.580886 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" event={"ID":"d57d79b3-a752-4808-9558-eef9087cd525","Type":"ContainerStarted","Data":"7371e1ea3576f33716991ef106afbec7a6a70a0ad2021209dde6908e1d50fd77"} Nov 01 22:26:34.581278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:34.580907 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:34.605607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:34.605549 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" podStartSLOduration=1.6055294949999999 podStartE2EDuration="1.605529495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:34.600298268 +0000 UTC m=+2541.574200822" watchObservedRunningTime="2026-11-01 22:26:34.605529495 +0000 UTC m=+2541.579432035" Nov 01 22:26:39.139887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.139852 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:26:39.146176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.146151 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.155861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.155834 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:26:39.176621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.176592 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-56452a-dockercfg-fscqg\"" Nov 01 22:26:39.226393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.205537 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-56452a\"" Nov 01 22:26:39.267073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.262557 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.267073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.262819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq522\" (UniqueName: \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.419118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.416734 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq522\" (UniqueName: \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.443258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.437576 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.463109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.459524 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.471206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.469273 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq522\" (UniqueName: \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\") pod \"payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.486677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.486327 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:39.860925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:39.860895 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:26:39.861498 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:26:39.861470 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865dd45e_fe43_439b_ab89_a44000605061.slice/crio-cf53c88730acc2ed39878b1457cb52598fda8c308660ba8c224ad065bbcfa3b2 WatchSource:0}: Error finding container cf53c88730acc2ed39878b1457cb52598fda8c308660ba8c224ad065bbcfa3b2: Status 404 returned error can't find the container with id cf53c88730acc2ed39878b1457cb52598fda8c308660ba8c224ad065bbcfa3b2 Nov 01 22:26:40.542009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.541978 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:26:40.542647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.542324 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" podUID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" containerName="payload-processing" containerID="cri-o://7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb" gracePeriod=30 Nov 01 22:26:40.545625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.545596 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:26:40.653247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.653209 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" podUID="071c0930-3c76-44d3-8492-7804ec29b763" containerName="payload-pre-processing" containerID="cri-o://73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b" gracePeriod=30 Nov 01 22:26:40.654373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.654340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" event={"ID":"865dd45e-fe43-439b-ab89-a44000605061","Type":"ContainerStarted","Data":"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37"} Nov 01 22:26:40.655447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.655419 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" event={"ID":"865dd45e-fe43-439b-ab89-a44000605061","Type":"ContainerStarted","Data":"cf53c88730acc2ed39878b1457cb52598fda8c308660ba8c224ad065bbcfa3b2"} Nov 01 22:26:40.655691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.655677 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:40.685096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.682068 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" podStartSLOduration=1.682050674 podStartE2EDuration="1.682050674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:40.677175197 +0000 UTC m=+2547.651077740" watchObservedRunningTime="2026-11-01 22:26:40.682050674 +0000 UTC m=+2547.655953213" Nov 01 22:26:40.724710 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.724369 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:26:40.725128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:40.724653 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" podUID="f7497356-bf15-45e6-b532-bab322220593" containerName="payload-pre-processing" containerID="cri-o://96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959" gracePeriod=30 Nov 01 22:26:41.074791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.074387 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:26:41.203324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.203243 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\" (UniqueName: \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\") pod \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " Nov 01 22:26:41.203562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.203545 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\") pod \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\" (UID: \"fd6a49b0-570d-47ce-8e4a-df8a5986a248\") " Nov 01 22:26:41.205399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.205375 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume" pod "fd6a49b0-570d-47ce-8e4a-df8a5986a248" (UID: "fd6a49b0-570d-47ce-8e4a-df8a5986a248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:41.235772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.234993 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs" pod "fd6a49b0-570d-47ce-8e4a-df8a5986a248" (UID: "fd6a49b0-570d-47ce-8e4a-df8a5986a248"). InnerVolumeSpecName "kube-api-access-nv2rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:41.335097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.335055 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2rs\" (UniqueName: \"kubernetes.io/projected/fd6a49b0-570d-47ce-8e4a-df8a5986a248-kube-api-access-nv2rs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.335097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.335099 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6a49b0-570d-47ce-8e4a-df8a5986a248-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.375672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.375195 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:26:41.390145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.389933 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:26:41.478336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.478228 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\") pod \"f7497356-bf15-45e6-b532-bab322220593\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " Nov 01 22:26:41.482740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.479326 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume" pod "f7497356-bf15-45e6-b532-bab322220593" (UID: "f7497356-bf15-45e6-b532-bab322220593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:41.583784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.583749 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\") pod \"071c0930-3c76-44d3-8492-7804ec29b763\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " Nov 01 22:26:41.584769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.584750 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\" (UniqueName: \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\") pod \"f7497356-bf15-45e6-b532-bab322220593\" (UID: \"f7497356-bf15-45e6-b532-bab322220593\") " Nov 01 22:26:41.585313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.585291 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\" (UniqueName: \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\") pod \"071c0930-3c76-44d3-8492-7804ec29b763\" (UID: \"071c0930-3c76-44d3-8492-7804ec29b763\") " Nov 01 22:26:41.586387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.586365 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7497356-bf15-45e6-b532-bab322220593-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.600264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.584670 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume" pod "071c0930-3c76-44d3-8492-7804ec29b763" (UID: "071c0930-3c76-44d3-8492-7804ec29b763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:41.645052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.645003 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj" pod "f7497356-bf15-45e6-b532-bab322220593" (UID: "f7497356-bf15-45e6-b532-bab322220593"). InnerVolumeSpecName "kube-api-access-l79hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:41.645649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.645630 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk" pod "071c0930-3c76-44d3-8492-7804ec29b763" (UID: "071c0930-3c76-44d3-8492-7804ec29b763"). InnerVolumeSpecName "kube-api-access-8qmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:41.672463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.672437 4395 generic.go:356] "Generic (PLEG): container finished" podID="f7497356-bf15-45e6-b532-bab322220593" containerID="96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959" exitCode=0 Nov 01 22:26:41.672783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.672753 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" event={"ID":"f7497356-bf15-45e6-b532-bab322220593","Type":"ContainerDied","Data":"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959"} Nov 01 22:26:41.672911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.672802 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" event={"ID":"f7497356-bf15-45e6-b532-bab322220593","Type":"ContainerDied","Data":"0deb9a8cd0a6414d26838c4b5b49727f504da577dcf7554024f1b1c32d0164bc"} Nov 01 22:26:41.672911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.672823 4395 scope.go:122] "RemoveContainer" containerID="96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959" Nov 01 22:26:41.674316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.674297 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m" Nov 01 22:26:41.676190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.675862 4395 generic.go:356] "Generic (PLEG): container finished" podID="071c0930-3c76-44d3-8492-7804ec29b763" containerID="73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b" exitCode=0 Nov 01 22:26:41.676190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.675927 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" Nov 01 22:26:41.676190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.676095 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" event={"ID":"071c0930-3c76-44d3-8492-7804ec29b763","Type":"ContainerDied","Data":"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b"} Nov 01 22:26:41.676190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.676141 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl" event={"ID":"071c0930-3c76-44d3-8492-7804ec29b763","Type":"ContainerDied","Data":"edc7b4e46afd97fb2dbfa442d4f44f7e2140693db804ad7f382cbe8d116364be"} Nov 01 22:26:41.686224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.685803 4395 generic.go:356] "Generic (PLEG): container finished" podID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" containerID="7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb" exitCode=0 Nov 01 22:26:41.686224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.686141 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" Nov 01 22:26:41.686224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.686211 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" event={"ID":"fd6a49b0-570d-47ce-8e4a-df8a5986a248","Type":"ContainerDied","Data":"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb"} Nov 01 22:26:41.686497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.686253 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2" event={"ID":"fd6a49b0-570d-47ce-8e4a-df8a5986a248","Type":"ContainerDied","Data":"c309cac664d3a84a5d3763d879cc139952984eb2b10ecd8a2c980e999d181bba"} Nov 01 22:26:41.693578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.693558 4395 scope.go:122] "RemoveContainer" containerID="96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959" Nov 01 22:26:41.693905 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:41.693880 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959\": container with ID starting with 96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959 not found: ID does not exist" containerID="96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959" Nov 01 22:26:41.693981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.693916 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959"} err="failed to get container status \"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959\": rpc error: code = NotFound desc = could not find container \"96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959\": container with ID starting with 96cf50d8721fa162c6bf12f4a908edd4fb526f15f2779fe6e35c445b2b32e959 not found: ID does not exist" Nov 01 22:26:41.693981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.693934 4395 scope.go:122] "RemoveContainer" containerID="73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b" Nov 01 22:26:41.700991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.700965 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmgk\" (UniqueName: \"kubernetes.io/projected/071c0930-3c76-44d3-8492-7804ec29b763-kube-api-access-8qmgk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.700991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.700989 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/071c0930-3c76-44d3-8492-7804ec29b763-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.701681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.701001 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79hj\" (UniqueName: \"kubernetes.io/projected/f7497356-bf15-45e6-b532-bab322220593-kube-api-access-l79hj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:41.707547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.707524 4395 scope.go:122] "RemoveContainer" containerID="73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b" Nov 01 22:26:41.707988 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:41.707958 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b\": container with ID starting with 73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b not found: ID does not exist" containerID="73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b" Nov 01 22:26:41.708074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.707995 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b"} err="failed to get container status \"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b\": rpc error: code = NotFound desc = could not find container \"73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b\": container with ID starting with 73316fea19693261d7dacfcb62b36a4828bfb0721cf06c2b7c615d88716ab97b not found: ID does not exist" Nov 01 22:26:41.708074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.708014 4395 scope.go:122] "RemoveContainer" containerID="7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb" Nov 01 22:26:41.719603 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.716627 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:26:41.721222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.721201 4395 scope.go:122] "RemoveContainer" containerID="7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb" Nov 01 22:26:41.721505 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:41.721480 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb\": container with ID starting with 7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb not found: ID does not exist" containerID="7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb" Nov 01 22:26:41.721570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.721516 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb"} err="failed to get container status \"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb\": rpc error: code = NotFound desc = could not find container \"7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb\": container with ID starting with 7a2a665f6ba31c330ec7736b5255ea8228f9c0a6e5e98a8ddc3d24fb56bd6eeb not found: ID does not exist" Nov 01 22:26:41.722860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.722837 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a355b-5cbb745987-tkhnl"] Nov 01 22:26:41.736912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.736851 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:26:41.742848 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.742823 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-5dd46f7c75-wbk4m"] Nov 01 22:26:41.755681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.755647 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:26:41.760341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:41.760314 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a355b-596cd847f6-h4xk2"] Nov 01 22:26:43.167921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:43.167889 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071c0930-3c76-44d3-8492-7804ec29b763" path="/var/lib/kubelet/pods/071c0930-3c76-44d3-8492-7804ec29b763/volumes" Nov 01 22:26:43.168337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:43.168319 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7497356-bf15-45e6-b532-bab322220593" path="/var/lib/kubelet/pods/f7497356-bf15-45e6-b532-bab322220593/volumes" Nov 01 22:26:43.168690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:43.168675 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" path="/var/lib/kubelet/pods/fd6a49b0-570d-47ce-8e4a-df8a5986a248/volumes" Nov 01 22:26:45.598967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:45.595646 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:26:48.388366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.388334 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:26:48.388847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.388586 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" containerID="cri-o://e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350" gracePeriod=30 Nov 01 22:26:48.971700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.971661 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:26:48.971963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.971931 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" containerID="cri-o://c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791" gracePeriod=30 Nov 01 22:26:48.978131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.978102 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:26:48.978516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.978489 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" containerID="cri-o://ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53" gracePeriod=30 Nov 01 22:26:48.994892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.994862 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:26:48.996158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:48.996126 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" containerID="cri-o://0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd" gracePeriod=30 Nov 01 22:26:51.588897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:51.588862 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:26:51.589373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:51.588923 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:26:52.312752 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:52.312567 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:26:52.700849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:52.700778 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:26:53.066974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.066885 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068102 4395 cpu_manager.go:414] "containerMap: removing container" podUID="071c0930-3c76-44d3-8492-7804ec29b763" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068125 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071c0930-3c76-44d3-8492-7804ec29b763" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068144 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" containerName="payload-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068151 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" containerName="payload-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068218 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f7497356-bf15-45e6-b532-bab322220593" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068227 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7497356-bf15-45e6-b532-bab322220593" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068413 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7497356-bf15-45e6-b532-bab322220593" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068432 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="071c0930-3c76-44d3-8492-7804ec29b763" containerName="payload-pre-processing" Nov 01 22:26:53.069205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.068443 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6a49b0-570d-47ce-8e4a-df8a5986a248" containerName="payload-processing" Nov 01 22:26:53.075714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.075688 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.075919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.075897 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:26:53.135146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135109 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135347 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135385 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135437 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135558 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135601 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwn4w\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.135734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.135707 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.237423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.237608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237561 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.237913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237710 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.237913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237741 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.237913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237830 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.238200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.237964 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.238200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.238003 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwn4w\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.238364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.238338 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.240698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.240673 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.240785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.240703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.240785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.240680 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.241378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.241356 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.249733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.249700 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.250481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.250463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwn4w\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dcjl7\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.407543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.407518 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.700958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.700928 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:26:53.706247 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:26:53.706222 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c936de_eaa8_4ca4_af0c_6634d535841a.slice/crio-cb1d5d50336a21b93434c80a1814266aa212cc6e2a54a1a57fc3575be639bd63 WatchSource:0}: Error finding container cb1d5d50336a21b93434c80a1814266aa212cc6e2a54a1a57fc3575be639bd63: Status 404 returned error can't find the container with id cb1d5d50336a21b93434c80a1814266aa212cc6e2a54a1a57fc3575be639bd63 Nov 01 22:26:53.706700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.706674 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:26:53.708975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.708949 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:26:53.709086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.709003 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:26:53.754711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.754685 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.754870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.754852 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.754958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.754895 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.754980 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755072 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755157 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755186 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755247 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\") pod \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\" (UID: \"3f9511b9-6ef3-4b00-9d5a-cfadc532745d\") " Nov 01 22:26:53.755418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.755347 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:53.756268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.756108 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.756640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.756428 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:53.756884 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.756772 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:53.757371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.757267 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:53.757371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.757333 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:53.759185 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.759159 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:53.759268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.759214 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "kube-api-access-mb7ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:53.760263 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.760241 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:53.760349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.760318 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy" pod "3f9511b9-6ef3-4b00-9d5a-cfadc532745d" (UID: "3f9511b9-6ef3-4b00-9d5a-cfadc532745d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:53.810728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.810698 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" event={"ID":"c8c936de-eaa8-4ca4-af0c-6634d535841a","Type":"ContainerStarted","Data":"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef"} Nov 01 22:26:53.810874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.810791 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:26:53.810874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.810811 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" event={"ID":"c8c936de-eaa8-4ca4-af0c-6634d535841a","Type":"ContainerStarted","Data":"cb1d5d50336a21b93434c80a1814266aa212cc6e2a54a1a57fc3575be639bd63"} Nov 01 22:26:53.812744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.812722 4395 generic.go:356] "Generic (PLEG): container finished" podID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerID="e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350" exitCode=0 Nov 01 22:26:53.812869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.812779 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" Nov 01 22:26:53.812869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.812806 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerDied","Data":"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350"} Nov 01 22:26:53.812869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.812861 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp" event={"ID":"3f9511b9-6ef3-4b00-9d5a-cfadc532745d","Type":"ContainerDied","Data":"5cece073f5303dcbbe7a3a42b2989cc9828249b2ed8fbab79e01a44590dcb779"} Nov 01 22:26:53.813041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.812886 4395 scope.go:122] "RemoveContainer" containerID="e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350" Nov 01 22:26:53.829420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.829400 4395 scope.go:122] "RemoveContainer" containerID="2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb" Nov 01 22:26:53.834181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.834133 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" podStartSLOduration=0.834116545 podStartE2EDuration="834.116545ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:53.832914587 +0000 UTC m=+2560.806817127" watchObservedRunningTime="2026-11-01 22:26:53.834116545 +0000 UTC m=+2560.808019085" Nov 01 22:26:53.844161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.844138 4395 scope.go:122] "RemoveContainer" containerID="e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350" Nov 01 22:26:53.844432 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:53.844407 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350\": container with ID starting with e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350 not found: ID does not exist" containerID="e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350" Nov 01 22:26:53.844506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.844440 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350"} err="failed to get container status \"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350\": rpc error: code = NotFound desc = could not find container \"e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350\": container with ID starting with e020b3625765b445c8c82e1d0c4980bd58665960fb526c60d46f3f03653b1350 not found: ID does not exist" Nov 01 22:26:53.844506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.844458 4395 scope.go:122] "RemoveContainer" containerID="2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb" Nov 01 22:26:53.844762 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:53.844733 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb\": container with ID starting with 2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb not found: ID does not exist" containerID="2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb" Nov 01 22:26:53.844838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.844769 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb"} err="failed to get container status \"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb\": rpc error: code = NotFound desc = could not find container \"2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb\": container with ID starting with 2f763ae287ef254058b21ae49608ba624d5b49ce9bc6e8d6a90fa77f87d6efcb not found: ID does not exist" Nov 01 22:26:53.856009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.855980 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:26:53.861020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.860996 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861040 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861055 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861070 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861084 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861098 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861113 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.861156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.861126 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7ws\" (UniqueName: \"kubernetes.io/projected/3f9511b9-6ef3-4b00-9d5a-cfadc532745d-kube-api-access-mb7ws\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:53.871354 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:53.871329 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a355b-openshift-default-8576bb9c45-9c7tp"] Nov 01 22:26:54.506112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.505949 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:26:54.540095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.538248 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:26:54.588607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.588184 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.589680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589647 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.589832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589715 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.589832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589766 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.590089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589837 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589883 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.590089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.589998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590061 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590102 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590142 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590231 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590273 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590302 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590349 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590444 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590489 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590566 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\") pod \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\" (UID: \"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3\") " Nov 01 22:26:54.590802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.590618 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\") pod \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\" (UID: \"21bbe5ce-b472-47f5-a567-7536d5d4ea02\") " Nov 01 22:26:54.594671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.592347 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.594671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.592572 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.599931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.599900 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:54.600248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.600221 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.602654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.602620 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.603353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.603330 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "kube-api-access-zj55k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.603694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.603677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.604002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.603985 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.641058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.626545 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.657616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.657586 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:54.711107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.671765 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735370 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735410 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735427 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735442 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735457 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735472 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735487 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735503 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj55k\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-kube-api-access-zj55k\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735519 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735534 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.779129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.735553 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.834755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.834682 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:54.834944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.834748 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.834944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.834890 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:54.835101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.835077 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.836859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.836832 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.837093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.837071 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf" pod "21bbe5ce-b472-47f5-a567-7536d5d4ea02" (UID: "21bbe5ce-b472-47f5-a567-7536d5d4ea02"). InnerVolumeSpecName "kube-api-access-25lrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.838395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838376 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.838522 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838510 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.838602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838592 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.838682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838671 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25lrf\" (UniqueName: \"kubernetes.io/projected/21bbe5ce-b472-47f5-a567-7536d5d4ea02-kube-api-access-25lrf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.838755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838746 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21bbe5ce-b472-47f5-a567-7536d5d4ea02-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.838842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.838829 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.839641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.839507 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy" pod "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" (UID: "30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.842101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.841368 4395 generic.go:356] "Generic (PLEG): container finished" podID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerID="ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53" exitCode=0 Nov 01 22:26:54.842101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.841469 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" event={"ID":"21bbe5ce-b472-47f5-a567-7536d5d4ea02","Type":"ContainerDied","Data":"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53"} Nov 01 22:26:54.842101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.841514 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" event={"ID":"21bbe5ce-b472-47f5-a567-7536d5d4ea02","Type":"ContainerDied","Data":"eca64b6394a54391d9cb18cdd6298892c9cc736126651a1cd71149bb38eedf65"} Nov 01 22:26:54.842101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.841541 4395 scope.go:122] "RemoveContainer" containerID="ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53" Nov 01 22:26:54.842101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.841640 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4" Nov 01 22:26:54.843866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.843789 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:26:54.851119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.850899 4395 generic.go:356] "Generic (PLEG): container finished" podID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerID="0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd" exitCode=0 Nov 01 22:26:54.851119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.850978 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" event={"ID":"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa","Type":"ContainerDied","Data":"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd"} Nov 01 22:26:54.857093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.854643 4395 generic.go:356] "Generic (PLEG): container finished" podID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerID="c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791" exitCode=0 Nov 01 22:26:54.857093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.855100 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" Nov 01 22:26:54.857643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.851021 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" event={"ID":"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa","Type":"ContainerDied","Data":"cd67356d2818bbe9e0c641ebc57e34d4c8675232c88c08b4f16c6961a740b89e"} Nov 01 22:26:54.857643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.857564 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" event={"ID":"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3","Type":"ContainerDied","Data":"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791"} Nov 01 22:26:54.857643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.857603 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl" event={"ID":"30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3","Type":"ContainerDied","Data":"3ed1c0fefbeca1df0d64d108a60ea1205bd97aa009dfa0321721c51408e3bfe7"} Nov 01 22:26:54.858842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.858525 4395 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dcjl7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:26:54.858842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.858579 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:26:54.872778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.871254 4395 scope.go:122] "RemoveContainer" containerID="ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53" Nov 01 22:26:54.873492 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:54.873468 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53\": container with ID starting with ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53 not found: ID does not exist" containerID="ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53" Nov 01 22:26:54.873639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.873616 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53"} err="failed to get container status \"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53\": rpc error: code = NotFound desc = could not find container \"ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53\": container with ID starting with ce5efd5aa94b4a2251a7afc59b677239834ac6fe3d5fe979c3432eb98ae79d53 not found: ID does not exist" Nov 01 22:26:54.873764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.873750 4395 scope.go:122] "RemoveContainer" containerID="0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd" Nov 01 22:26:54.902141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.898693 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:26:54.906749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.905992 4395 scope.go:122] "RemoveContainer" containerID="0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd" Nov 01 22:26:54.906749 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:54.906381 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd\": container with ID starting with 0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd not found: ID does not exist" containerID="0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd" Nov 01 22:26:54.906749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.906414 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd"} err="failed to get container status \"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd\": rpc error: code = NotFound desc = could not find container \"0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd\": container with ID starting with 0059e4e5a113cd7dce98f84edfaa0ad0bd2c3cb724ee854aff8e35757e8eeadd not found: ID does not exist" Nov 01 22:26:54.906749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.906438 4395 scope.go:122] "RemoveContainer" containerID="c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791" Nov 01 22:26:54.908704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.908689 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-pffq4"] Nov 01 22:26:54.924395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.924368 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:26:54.930077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.930055 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-nt4hl"] Nov 01 22:26:54.932642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.931186 4395 scope.go:122] "RemoveContainer" containerID="c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791" Nov 01 22:26:54.932642 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:26:54.931648 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791\": container with ID starting with c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791 not found: ID does not exist" containerID="c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791" Nov 01 22:26:54.932642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.931734 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791"} err="failed to get container status \"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791\": rpc error: code = NotFound desc = could not find container \"c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791\": container with ID starting with c7678eb2ba447269e1cbff0360c72930d6ea001ed285099b2999bf3981c8a791 not found: ID does not exist" Nov 01 22:26:54.939988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.939960 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940090 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940139 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940238 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940375 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940415 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940469 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940532 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940577 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\") pod \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\" (UID: \"6a8e6cb3-854f-4431-9e03-4cfcc5f03daa\") " Nov 01 22:26:54.940811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.940785 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.942911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.942875 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.943089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.942917 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:54.943850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.943804 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.945488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.945462 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.946647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.946621 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.947200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.947177 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:26:54.958862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.958835 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "kube-api-access-8xk9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:54.961901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.961862 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:54.962159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.962142 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:26:54.964304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:54.964274 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token" pod "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" (UID: "6a8e6cb3-854f-4431-9e03-4cfcc5f03daa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044305 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044334 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044345 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044354 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044364 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk9w\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-kube-api-access-8xk9w\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044373 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044381 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.044747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.044389 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:55.239232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.239185 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" path="/var/lib/kubelet/pods/21bbe5ce-b472-47f5-a567-7536d5d4ea02/volumes" Nov 01 22:26:55.243854 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.243815 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" path="/var/lib/kubelet/pods/30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3/volumes" Nov 01 22:26:55.244754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.244729 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" path="/var/lib/kubelet/pods/3f9511b9-6ef3-4b00-9d5a-cfadc532745d/volumes" Nov 01 22:26:55.865425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.865395 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8" Nov 01 22:26:55.891926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.891897 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:26:55.897757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:55.897732 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-t55x8"] Nov 01 22:26:57.167955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:57.167922 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" path="/var/lib/kubelet/pods/6a8e6cb3-854f-4431-9e03-4cfcc5f03daa/volumes" Nov 01 22:26:57.856909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:26:57.856883 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:27:03.440282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:03.440254 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:27:03.456123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:03.456096 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:27:06.025278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.025242 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:27:06.025835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.025448 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" podUID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" containerName="payload-processing" containerID="cri-o://b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371" gracePeriod=30 Nov 01 22:27:06.035165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.035125 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:27:06.035377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.035318 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" podUID="b90f0c03-589b-4a47-84d3-257348fa517f" containerName="payload-pre-processing" containerID="cri-o://393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5" gracePeriod=30 Nov 01 22:27:06.617771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.617702 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:27:06.631173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.631058 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\" (UniqueName: \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\") pod \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " Nov 01 22:27:06.631173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.631106 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\") pod \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\" (UID: \"9ec612db-5ae2-477e-84bc-71a5239b8a5c\") " Nov 01 22:27:06.631550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.631501 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume" pod "9ec612db-5ae2-477e-84bc-71a5239b8a5c" (UID: "9ec612db-5ae2-477e-84bc-71a5239b8a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:06.631905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.631863 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec612db-5ae2-477e-84bc-71a5239b8a5c-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:06.633549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.633518 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf" pod "9ec612db-5ae2-477e-84bc-71a5239b8a5c" (UID: "9ec612db-5ae2-477e-84bc-71a5239b8a5c"). InnerVolumeSpecName "kube-api-access-68vdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:06.702477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.702372 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:27:06.732490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.732456 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\" (UniqueName: \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\") pod \"b90f0c03-589b-4a47-84d3-257348fa517f\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " Nov 01 22:27:06.732665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.732524 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\") pod \"b90f0c03-589b-4a47-84d3-257348fa517f\" (UID: \"b90f0c03-589b-4a47-84d3-257348fa517f\") " Nov 01 22:27:06.733073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.733036 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume" pod "b90f0c03-589b-4a47-84d3-257348fa517f" (UID: "b90f0c03-589b-4a47-84d3-257348fa517f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:06.733311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.733268 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90f0c03-589b-4a47-84d3-257348fa517f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:06.733311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.733294 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vdf\" (UniqueName: \"kubernetes.io/projected/9ec612db-5ae2-477e-84bc-71a5239b8a5c-kube-api-access-68vdf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:06.735170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.735142 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp" pod "b90f0c03-589b-4a47-84d3-257348fa517f" (UID: "b90f0c03-589b-4a47-84d3-257348fa517f"). InnerVolumeSpecName "kube-api-access-sfxlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:06.834824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.834792 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxlp\" (UniqueName: \"kubernetes.io/projected/b90f0c03-589b-4a47-84d3-257348fa517f-kube-api-access-sfxlp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:06.979598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.979500 4395 generic.go:356] "Generic (PLEG): container finished" podID="b90f0c03-589b-4a47-84d3-257348fa517f" containerID="393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5" exitCode=0 Nov 01 22:27:06.979598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.979548 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" Nov 01 22:27:06.979598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.979567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" event={"ID":"b90f0c03-589b-4a47-84d3-257348fa517f","Type":"ContainerDied","Data":"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5"} Nov 01 22:27:06.979885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.979627 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd" event={"ID":"b90f0c03-589b-4a47-84d3-257348fa517f","Type":"ContainerDied","Data":"ff3d57569ca15a2783e20d5d6d05de9d7b11e5713234e70a527d39c75ad1b379"} Nov 01 22:27:06.979885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.979643 4395 scope.go:122] "RemoveContainer" containerID="393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5" Nov 01 22:27:06.981650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.981618 4395 generic.go:356] "Generic (PLEG): container finished" podID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" containerID="b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371" exitCode=0 Nov 01 22:27:06.981793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.981673 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" Nov 01 22:27:06.981793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.981759 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" event={"ID":"9ec612db-5ae2-477e-84bc-71a5239b8a5c","Type":"ContainerDied","Data":"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371"} Nov 01 22:27:06.981913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.981800 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg" event={"ID":"9ec612db-5ae2-477e-84bc-71a5239b8a5c","Type":"ContainerDied","Data":"b423fa8840e7375960dada274fd5727ad3442452bd385b42214c721c19a7feaa"} Nov 01 22:27:06.994662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.994641 4395 scope.go:122] "RemoveContainer" containerID="393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5" Nov 01 22:27:06.994949 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:06.994928 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5\": container with ID starting with 393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5 not found: ID does not exist" containerID="393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5" Nov 01 22:27:06.995089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.994960 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5"} err="failed to get container status \"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5\": rpc error: code = NotFound desc = could not find container \"393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5\": container with ID starting with 393ec03d52fe5d4bd14b2e6be72323a70d2fd6328ebd5d2edff7debf380c03a5 not found: ID does not exist" Nov 01 22:27:06.995089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:06.994986 4395 scope.go:122] "RemoveContainer" containerID="b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371" Nov 01 22:27:07.004997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.004979 4395 scope.go:122] "RemoveContainer" containerID="b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371" Nov 01 22:27:07.005292 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:07.005269 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371\": container with ID starting with b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371 not found: ID does not exist" containerID="b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371" Nov 01 22:27:07.005353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.005301 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371"} err="failed to get container status \"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371\": rpc error: code = NotFound desc = could not find container \"b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371\": container with ID starting with b53c6a819341b29d8137d5ddd7ca9265deb03be330a423668dd3b9e3131e8371 not found: ID does not exist" Nov 01 22:27:07.047520 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.047489 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:27:07.098410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.098387 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-829cb0-6b7f95ffdd-qh4xg"] Nov 01 22:27:07.123087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.122801 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:27:07.127130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.127104 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-9fc54866c-hgmhd"] Nov 01 22:27:07.167865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.167840 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" path="/var/lib/kubelet/pods/9ec612db-5ae2-477e-84bc-71a5239b8a5c/volumes" Nov 01 22:27:07.168344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.168329 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90f0c03-589b-4a47-84d3-257348fa517f" path="/var/lib/kubelet/pods/b90f0c03-589b-4a47-84d3-257348fa517f/volumes" Nov 01 22:27:07.323961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.323930 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:27:07.325412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.324868 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" podUID="f9fe3b52-ca4e-453f-a309-682e457da54a" containerName="maas-api" containerID="cri-o://ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67" gracePeriod=30 Nov 01 22:27:07.404723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.404690 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:27:07.404934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:07.404912 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" podUID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" containerName="payload-processing" containerID="cri-o://e2a30f81fe7b9b581c0c7791d7242e5a150a539249eefbcc756ff5b4c2b9bc47" gracePeriod=30 Nov 01 22:27:08.026824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.026568 4395 generic.go:356] "Generic (PLEG): container finished" podID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" containerID="e2a30f81fe7b9b581c0c7791d7242e5a150a539249eefbcc756ff5b4c2b9bc47" exitCode=0 Nov 01 22:27:08.027979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.026764 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" event={"ID":"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf","Type":"ContainerDied","Data":"e2a30f81fe7b9b581c0c7791d7242e5a150a539249eefbcc756ff5b4c2b9bc47"} Nov 01 22:27:08.030711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.030687 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:27:08.032745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.032718 4395 generic.go:356] "Generic (PLEG): container finished" podID="f9fe3b52-ca4e-453f-a309-682e457da54a" containerID="ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67" exitCode=0 Nov 01 22:27:08.032853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.032842 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" event={"ID":"f9fe3b52-ca4e-453f-a309-682e457da54a","Type":"ContainerDied","Data":"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67"} Nov 01 22:27:08.033009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.032890 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" event={"ID":"f9fe3b52-ca4e-453f-a309-682e457da54a","Type":"ContainerDied","Data":"4c7f7a71affcd4296627a623e74fbe504077db35b2b2bbe89e95ce2911da7cb0"} Nov 01 22:27:08.033009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.032923 4395 scope.go:122] "RemoveContainer" containerID="ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67" Nov 01 22:27:08.045559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.045533 4395 scope.go:122] "RemoveContainer" containerID="ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67" Nov 01 22:27:08.046219 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:08.046192 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67\": container with ID starting with ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67 not found: ID does not exist" containerID="ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67" Nov 01 22:27:08.046343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.046229 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67"} err="failed to get container status \"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67\": rpc error: code = NotFound desc = could not find container \"ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67\": container with ID starting with ba754b8393c285b6d92e698e2f5b0c11cd52cf4855fc4f2342a2200e58cc5d67 not found: ID does not exist" Nov 01 22:27:08.047413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.047394 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:27:08.107264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107228 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\" (UniqueName: \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\") pod \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " Nov 01 22:27:08.107764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107391 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\") pod \"f9fe3b52-ca4e-453f-a309-682e457da54a\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " Nov 01 22:27:08.107764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107497 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\") pod \"f9fe3b52-ca4e-453f-a309-682e457da54a\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " Nov 01 22:27:08.107764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107575 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\") pod \"f9fe3b52-ca4e-453f-a309-682e457da54a\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " Nov 01 22:27:08.107764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107609 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\") pod \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\" (UID: \"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf\") " Nov 01 22:27:08.107764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.107700 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\" (UniqueName: \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\") pod \"f9fe3b52-ca4e-453f-a309-682e457da54a\" (UID: \"f9fe3b52-ca4e-453f-a309-682e457da54a\") " Nov 01 22:27:08.108509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.108476 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca" pod "f9fe3b52-ca4e-453f-a309-682e457da54a" (UID: "f9fe3b52-ca4e-453f-a309-682e457da54a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:08.108509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.108489 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume" pod "b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" (UID: "b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:08.111498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.111467 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls" pod "f9fe3b52-ca4e-453f-a309-682e457da54a" (UID: "f9fe3b52-ca4e-453f-a309-682e457da54a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:08.111498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.111490 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs" pod "f9fe3b52-ca4e-453f-a309-682e457da54a" (UID: "f9fe3b52-ca4e-453f-a309-682e457da54a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:08.111657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.111537 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4" pod "f9fe3b52-ca4e-453f-a309-682e457da54a" (UID: "f9fe3b52-ca4e-453f-a309-682e457da54a"). InnerVolumeSpecName "kube-api-access-dk5m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:08.111657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.111604 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb" pod "b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" (UID: "b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf"). InnerVolumeSpecName "kube-api-access-9tjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:08.210412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210379 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.210412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210410 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe3b52-ca4e-453f-a309-682e457da54a-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.210648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210424 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe3b52-ca4e-453f-a309-682e457da54a-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.210648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210438 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.210648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210455 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5m4\" (UniqueName: \"kubernetes.io/projected/f9fe3b52-ca4e-453f-a309-682e457da54a-kube-api-access-dk5m4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.210648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.210468 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjvb\" (UniqueName: \"kubernetes.io/projected/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf-kube-api-access-9tjvb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.441411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.441320 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" podUID="cad9d39c-e5bf-4279-b9dd-0459084442a9" containerName="payload-processing" containerID="cri-o://8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7" gracePeriod=25 Nov 01 22:27:08.456687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.456641 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" podUID="14cfab37-10eb-4ac8-9972-45366c8098d0" containerName="payload-pre-processing" containerID="cri-o://c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f" gracePeriod=25 Nov 01 22:27:08.727628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.727182 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:27:08.780388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.780363 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:27:08.817914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.817877 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\" (UniqueName: \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\") pod \"14cfab37-10eb-4ac8-9972-45366c8098d0\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " Nov 01 22:27:08.818122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.817955 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\") pod \"14cfab37-10eb-4ac8-9972-45366c8098d0\" (UID: \"14cfab37-10eb-4ac8-9972-45366c8098d0\") " Nov 01 22:27:08.818122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.818046 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\") pod \"cad9d39c-e5bf-4279-b9dd-0459084442a9\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " Nov 01 22:27:08.818453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.818330 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\" (UniqueName: \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\") pod \"cad9d39c-e5bf-4279-b9dd-0459084442a9\" (UID: \"cad9d39c-e5bf-4279-b9dd-0459084442a9\") " Nov 01 22:27:08.818566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.818489 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume" pod "cad9d39c-e5bf-4279-b9dd-0459084442a9" (UID: "cad9d39c-e5bf-4279-b9dd-0459084442a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:08.819250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.819222 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad9d39c-e5bf-4279-b9dd-0459084442a9-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.819250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.819239 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume" pod "14cfab37-10eb-4ac8-9972-45366c8098d0" (UID: "14cfab37-10eb-4ac8-9972-45366c8098d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:08.820613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.820590 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7" pod "14cfab37-10eb-4ac8-9972-45366c8098d0" (UID: "14cfab37-10eb-4ac8-9972-45366c8098d0"). InnerVolumeSpecName "kube-api-access-nprm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:08.821229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.821209 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k" pod "cad9d39c-e5bf-4279-b9dd-0459084442a9" (UID: "cad9d39c-e5bf-4279-b9dd-0459084442a9"). InnerVolumeSpecName "kube-api-access-kkk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:08.920878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.920841 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprm7\" (UniqueName: \"kubernetes.io/projected/14cfab37-10eb-4ac8-9972-45366c8098d0-kube-api-access-nprm7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.920878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.920879 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cfab37-10eb-4ac8-9972-45366c8098d0-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:08.921107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:08.920892 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkk5k\" (UniqueName: \"kubernetes.io/projected/cad9d39c-e5bf-4279-b9dd-0459084442a9-kube-api-access-kkk5k\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:09.042164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.042126 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq" Nov 01 22:27:09.044308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.044255 4395 generic.go:356] "Generic (PLEG): container finished" podID="14cfab37-10eb-4ac8-9972-45366c8098d0" containerID="c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f" exitCode=0 Nov 01 22:27:09.044308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.044294 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" Nov 01 22:27:09.044558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.044314 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" event={"ID":"14cfab37-10eb-4ac8-9972-45366c8098d0","Type":"ContainerDied","Data":"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f"} Nov 01 22:27:09.044558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.044366 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx" event={"ID":"14cfab37-10eb-4ac8-9972-45366c8098d0","Type":"ContainerDied","Data":"eb09b9ab41c2171fc112b3ac12ede73423aa4fa87220ccddb9f577dcbdef5330"} Nov 01 22:27:09.044558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.044395 4395 scope.go:122] "RemoveContainer" containerID="c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f" Nov 01 22:27:09.047499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.047466 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" Nov 01 22:27:09.047607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.047554 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75" event={"ID":"b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf","Type":"ContainerDied","Data":"d79b34960afb1af4ca6504ef30a5e30e148ce8c73a34ff5dcd377e0bf19b2c49"} Nov 01 22:27:09.050155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.050129 4395 generic.go:356] "Generic (PLEG): container finished" podID="cad9d39c-e5bf-4279-b9dd-0459084442a9" containerID="8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7" exitCode=0 Nov 01 22:27:09.050265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.050195 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" event={"ID":"cad9d39c-e5bf-4279-b9dd-0459084442a9","Type":"ContainerDied","Data":"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7"} Nov 01 22:27:09.050265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.050233 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" event={"ID":"cad9d39c-e5bf-4279-b9dd-0459084442a9","Type":"ContainerDied","Data":"cee2c87b83e71950e58eba0dee6321990eb1c1a7db45ce54ea902162cd9f48b3"} Nov 01 22:27:09.050383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.050301 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b" Nov 01 22:27:09.082283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.082264 4395 scope.go:122] "RemoveContainer" containerID="c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f" Nov 01 22:27:09.082657 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:09.082637 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f\": container with ID starting with c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f not found: ID does not exist" containerID="c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f" Nov 01 22:27:09.082755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.082664 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f"} err="failed to get container status \"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f\": rpc error: code = NotFound desc = could not find container \"c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f\": container with ID starting with c14ea570564ef05610a52c026ad9f3b79b05de67e641e13f964f4857c141b89f not found: ID does not exist" Nov 01 22:27:09.082755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.082684 4395 scope.go:122] "RemoveContainer" containerID="e2a30f81fe7b9b581c0c7791d7242e5a150a539249eefbcc756ff5b4c2b9bc47" Nov 01 22:27:09.093304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.093283 4395 scope.go:122] "RemoveContainer" containerID="8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7" Nov 01 22:27:09.102983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.102960 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:27:09.104707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.104687 4395 scope.go:122] "RemoveContainer" containerID="8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7" Nov 01 22:27:09.105051 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:09.105012 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7\": container with ID starting with 8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7 not found: ID does not exist" containerID="8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7" Nov 01 22:27:09.105206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.105057 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7"} err="failed to get container status \"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7\": rpc error: code = NotFound desc = could not find container \"8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7\": container with ID starting with 8208ca644d0a32baddb125f9df558e4c7263908ddd2af1308f9296fb686adbf7 not found: ID does not exist" Nov 01 22:27:09.108959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.108928 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad047b-66fdf5c576-2zw75"] Nov 01 22:27:09.121290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.121267 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:27:09.137756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.125399 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad047b-7dff68698c-w62qq"] Nov 01 22:27:09.223758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.223723 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" path="/var/lib/kubelet/pods/b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf/volumes" Nov 01 22:27:09.224394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.224372 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9fe3b52-ca4e-453f-a309-682e457da54a" path="/var/lib/kubelet/pods/f9fe3b52-ca4e-453f-a309-682e457da54a/volumes" Nov 01 22:27:09.224769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.224750 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:27:09.224847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.224791 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-668fb87544-crv7b"] Nov 01 22:27:09.229132 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.229107 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:27:09.232951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:09.232927 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-57fc7fcd7f-rj7jx"] Nov 01 22:27:11.167307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:11.167278 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cfab37-10eb-4ac8-9972-45366c8098d0" path="/var/lib/kubelet/pods/14cfab37-10eb-4ac8-9972-45366c8098d0/volumes" Nov 01 22:27:11.167741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:11.167709 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad9d39c-e5bf-4279-b9dd-0459084442a9" path="/var/lib/kubelet/pods/cad9d39c-e5bf-4279-b9dd-0459084442a9/volumes" Nov 01 22:27:21.385343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.385295 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:27:21.387011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.386976 4395 cpu_manager.go:414] "containerMap: removing container" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" Nov 01 22:27:21.387011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387003 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387051 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387060 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387072 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" containerName="payload-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387080 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" containerName="payload-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387092 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f9fe3b52-ca4e-453f-a309-682e457da54a" containerName="maas-api" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387098 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fe3b52-ca4e-453f-a309-682e457da54a" containerName="maas-api" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387107 4395 cpu_manager.go:414] "containerMap: removing container" podUID="14cfab37-10eb-4ac8-9972-45366c8098d0" containerName="payload-pre-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387112 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cfab37-10eb-4ac8-9972-45366c8098d0" containerName="payload-pre-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387125 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387130 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387142 4395 cpu_manager.go:414] "containerMap: removing container" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387147 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387159 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cad9d39c-e5bf-4279-b9dd-0459084442a9" containerName="payload-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387163 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad9d39c-e5bf-4279-b9dd-0459084442a9" containerName="payload-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387174 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387178 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387190 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" containerName="payload-processing" Nov 01 22:27:21.387193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387195 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" containerName="payload-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387213 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b90f0c03-589b-4a47-84d3-257348fa517f" containerName="payload-pre-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387217 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90f0c03-589b-4a47-84d3-257348fa517f" containerName="payload-pre-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387341 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8aeb3fe-7b12-47f8-a18c-a14f9b2090cf" containerName="payload-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387352 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90f0c03-589b-4a47-84d3-257348fa517f" containerName="payload-pre-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387363 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d6c0a3-4a25-4be0-9f1c-04f5f2ef95e3" containerName="istio-proxy" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387369 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387374 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec612db-5ae2-477e-84bc-71a5239b8a5c" containerName="payload-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387379 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fe3b52-ca4e-453f-a309-682e457da54a" containerName="maas-api" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387386 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cfab37-10eb-4ac8-9972-45366c8098d0" containerName="payload-pre-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387392 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9511b9-6ef3-4b00-9d5a-cfadc532745d" containerName="istio-proxy" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387399 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bbe5ce-b472-47f5-a567-7536d5d4ea02" containerName="istio-proxy" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387405 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad9d39c-e5bf-4279-b9dd-0459084442a9" containerName="payload-processing" Nov 01 22:27:21.387993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.387410 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8e6cb3-854f-4431-9e03-4cfcc5f03daa" containerName="istio-proxy" Nov 01 22:27:21.413274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.413245 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:27:21.413439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.413380 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.422551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.422055 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a00c6a-dockercfg-84dnl\"" Nov 01 22:27:21.428811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.428772 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a00c6a\"" Nov 01 22:27:21.499850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.499819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.500092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.500065 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8d4r\" (UniqueName: \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.601834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.601793 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.602081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.602057 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8d4r\" (UniqueName: \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.602822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.602793 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.616773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.616737 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8d4r\" (UniqueName: \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\") pod \"payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:21.732850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:21.732770 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:22.101533 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:27:22.101492 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff121c3_177d_4a67_8f13_98da0bde9334.slice/crio-2d74151a25b726274bc5c010de97ad2586515334996bc70f4beab18b52c69db3 WatchSource:0}: Error finding container 2d74151a25b726274bc5c010de97ad2586515334996bc70f4beab18b52c69db3: Status 404 returned error can't find the container with id 2d74151a25b726274bc5c010de97ad2586515334996bc70f4beab18b52c69db3 Nov 01 22:27:22.106834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.106567 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:27:22.263789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.263753 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" event={"ID":"7ff121c3-177d-4a67-8f13-98da0bde9334","Type":"ContainerStarted","Data":"acc9e533505146d5d850cc7400820b80d9b271a678f837d1ab3cacc66954faf2"} Nov 01 22:27:22.263946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.263812 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:22.263946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.263829 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" event={"ID":"7ff121c3-177d-4a67-8f13-98da0bde9334","Type":"ContainerStarted","Data":"2d74151a25b726274bc5c010de97ad2586515334996bc70f4beab18b52c69db3"} Nov 01 22:27:22.282936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.282870 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" podStartSLOduration=1.282851434 podStartE2EDuration="1.282851434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:22.281620282 +0000 UTC m=+2589.255522822" watchObservedRunningTime="2026-11-01 22:27:22.282851434 +0000 UTC m=+2589.256753974" Nov 01 22:27:22.313956 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:22.313922 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:27:22.725044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:22.724996 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:27:24.878909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:24.878867 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:27:25.795321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:25.795288 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:27:25.814992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:25.814958 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:27:27.726298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:27.726264 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" podUID="d57d79b3-a752-4808-9558-eef9087cd525" containerName="payload-pre-processing" containerID="cri-o://3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289" gracePeriod=25 Nov 01 22:27:28.040275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.040248 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:27:28.182247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.182196 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\" (UniqueName: \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\") pod \"d57d79b3-a752-4808-9558-eef9087cd525\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " Nov 01 22:27:28.182457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.182372 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\") pod \"d57d79b3-a752-4808-9558-eef9087cd525\" (UID: \"d57d79b3-a752-4808-9558-eef9087cd525\") " Nov 01 22:27:28.182823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.182791 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume" pod "d57d79b3-a752-4808-9558-eef9087cd525" (UID: "d57d79b3-a752-4808-9558-eef9087cd525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:28.184633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.184607 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr" pod "d57d79b3-a752-4808-9558-eef9087cd525" (UID: "d57d79b3-a752-4808-9558-eef9087cd525"). InnerVolumeSpecName "kube-api-access-qshqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:28.286227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.286180 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57d79b3-a752-4808-9558-eef9087cd525-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:28.286533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.286510 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshqr\" (UniqueName: \"kubernetes.io/projected/d57d79b3-a752-4808-9558-eef9087cd525-kube-api-access-qshqr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:28.399656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.399623 4395 generic.go:356] "Generic (PLEG): container finished" podID="d57d79b3-a752-4808-9558-eef9087cd525" containerID="3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289" exitCode=0 Nov 01 22:27:28.399821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.399672 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" Nov 01 22:27:28.399821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.399714 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" event={"ID":"d57d79b3-a752-4808-9558-eef9087cd525","Type":"ContainerDied","Data":"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289"} Nov 01 22:27:28.399821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.399775 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw" event={"ID":"d57d79b3-a752-4808-9558-eef9087cd525","Type":"ContainerDied","Data":"7371e1ea3576f33716991ef106afbec7a6a70a0ad2021209dde6908e1d50fd77"} Nov 01 22:27:28.399821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.399803 4395 scope.go:122] "RemoveContainer" containerID="3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289" Nov 01 22:27:28.411914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.411893 4395 scope.go:122] "RemoveContainer" containerID="3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289" Nov 01 22:27:28.412219 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:28.412199 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289\": container with ID starting with 3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289 not found: ID does not exist" containerID="3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289" Nov 01 22:27:28.412306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.412228 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289"} err="failed to get container status \"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289\": rpc error: code = NotFound desc = could not find container \"3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289\": container with ID starting with 3b51c5d05bc5126e1c0c1802c3e13dcb97cb9dfcc9d6e46b3ab35cef0e004289 not found: ID does not exist" Nov 01 22:27:28.426907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.424212 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:27:28.428178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.428159 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-85b847957f-k5xnw"] Nov 01 22:27:28.554002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:28.553945 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:27:29.177165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:29.177133 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57d79b3-a752-4808-9558-eef9087cd525" path="/var/lib/kubelet/pods/d57d79b3-a752-4808-9558-eef9087cd525/volumes" Nov 01 22:27:29.883089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:29.882093 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" containerID="cri-o://2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc" gracePeriod=25 Nov 01 22:27:30.795959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:30.795921 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" containerID="cri-o://fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761" gracePeriod=25 Nov 01 22:27:30.815755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:30.815726 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" containerID="cri-o://f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f" gracePeriod=25 Nov 01 22:27:33.334949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:33.334921 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:27:33.368230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:33.368190 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:33.554293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:33.554236 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" podUID="865dd45e-fe43-439b-ab89-a44000605061" containerName="payload-pre-processing" containerID="cri-o://830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37" gracePeriod=25 Nov 01 22:27:33.910190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:33.910163 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:27:34.012310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.012279 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\" (UniqueName: \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\") pod \"865dd45e-fe43-439b-ab89-a44000605061\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " Nov 01 22:27:34.012472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.012333 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\") pod \"865dd45e-fe43-439b-ab89-a44000605061\" (UID: \"865dd45e-fe43-439b-ab89-a44000605061\") " Nov 01 22:27:34.012772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.012744 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume" pod "865dd45e-fe43-439b-ab89-a44000605061" (UID: "865dd45e-fe43-439b-ab89-a44000605061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:34.013093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.013074 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865dd45e-fe43-439b-ab89-a44000605061-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:34.014658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.014637 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522" pod "865dd45e-fe43-439b-ab89-a44000605061" (UID: "865dd45e-fe43-439b-ab89-a44000605061"). InnerVolumeSpecName "kube-api-access-kq522". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:34.114805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.114772 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq522\" (UniqueName: \"kubernetes.io/projected/865dd45e-fe43-439b-ab89-a44000605061-kube-api-access-kq522\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:34.500536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.500496 4395 generic.go:356] "Generic (PLEG): container finished" podID="865dd45e-fe43-439b-ab89-a44000605061" containerID="830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37" exitCode=0 Nov 01 22:27:34.500964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.500675 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" event={"ID":"865dd45e-fe43-439b-ab89-a44000605061","Type":"ContainerDied","Data":"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37"} Nov 01 22:27:34.500964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.500739 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" event={"ID":"865dd45e-fe43-439b-ab89-a44000605061","Type":"ContainerDied","Data":"cf53c88730acc2ed39878b1457cb52598fda8c308660ba8c224ad065bbcfa3b2"} Nov 01 22:27:34.500964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.500769 4395 scope.go:122] "RemoveContainer" containerID="830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37" Nov 01 22:27:34.500964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.500923 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6" Nov 01 22:27:34.528273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.528239 4395 scope.go:122] "RemoveContainer" containerID="830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37" Nov 01 22:27:34.529354 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:34.529275 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37\": container with ID starting with 830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37 not found: ID does not exist" containerID="830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37" Nov 01 22:27:34.529484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.529366 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37"} err="failed to get container status \"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37\": rpc error: code = NotFound desc = could not find container \"830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37\": container with ID starting with 830d835e7e072e98af76c629bce01f3bd3cd0d818e7c43e40f18f78245cafb37 not found: ID does not exist" Nov 01 22:27:34.545234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.545206 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:27:34.549211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:34.549186 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-95b5c7675-td2g6"] Nov 01 22:27:35.170967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:35.170933 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865dd45e-fe43-439b-ab89-a44000605061" path="/var/lib/kubelet/pods/865dd45e-fe43-439b-ab89-a44000605061/volumes" Nov 01 22:27:38.006853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:38.006808 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:38.007314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:38.006973 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:43.369436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:43.369394 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:47.875593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:47.875562 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:27:47.890143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:47.890110 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:27:48.007442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:48.007407 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:48.007656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:48.007464 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:50.079603 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.079578 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:27:50.196360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.196260 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\" (UniqueName: \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\") pod \"ba346d51-88f9-468a-817e-d5963a8165e2\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " Nov 01 22:27:50.196532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.196371 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\") pod \"ba346d51-88f9-468a-817e-d5963a8165e2\" (UID: \"ba346d51-88f9-468a-817e-d5963a8165e2\") " Nov 01 22:27:50.196765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.196735 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume" pod "ba346d51-88f9-468a-817e-d5963a8165e2" (UID: "ba346d51-88f9-468a-817e-d5963a8165e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:50.197043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.197008 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba346d51-88f9-468a-817e-d5963a8165e2-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:50.198888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.198862 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp" pod "ba346d51-88f9-468a-817e-d5963a8165e2" (UID: "ba346d51-88f9-468a-817e-d5963a8165e2"). InnerVolumeSpecName "kube-api-access-l2vqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:50.298277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.298243 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2vqp\" (UniqueName: \"kubernetes.io/projected/ba346d51-88f9-468a-817e-d5963a8165e2-kube-api-access-l2vqp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:50.675292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.675259 4395 generic.go:356] "Generic (PLEG): container finished" podID="ba346d51-88f9-468a-817e-d5963a8165e2" containerID="2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc" exitCode=0 Nov 01 22:27:50.675478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.675387 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" Nov 01 22:27:50.675478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.675398 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" event={"ID":"ba346d51-88f9-468a-817e-d5963a8165e2","Type":"ContainerDied","Data":"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc"} Nov 01 22:27:50.675577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.675496 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt" event={"ID":"ba346d51-88f9-468a-817e-d5963a8165e2","Type":"ContainerDied","Data":"fc3a11d045098675689eca975e60ad8731926589a880b01579c3b9f9dbd26480"} Nov 01 22:27:50.675577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.675526 4395 scope.go:122] "RemoveContainer" containerID="2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc" Nov 01 22:27:50.686622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.686605 4395 scope.go:122] "RemoveContainer" containerID="2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc" Nov 01 22:27:50.686888 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:50.686871 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc\": container with ID starting with 2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc not found: ID does not exist" containerID="2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc" Nov 01 22:27:50.686944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.686895 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc"} err="failed to get container status \"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc\": rpc error: code = NotFound desc = could not find container \"2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc\": container with ID starting with 2d6457ca12ea35160df05c6b59a9360173bcfb84f0e910e64f4bdd6a4afb0edc not found: ID does not exist" Nov 01 22:27:50.700059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.700038 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:27:50.707403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:50.707377 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-702002-768fc75ff4-pgbkt"] Nov 01 22:27:51.171368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.171337 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" path="/var/lib/kubelet/pods/ba346d51-88f9-468a-817e-d5963a8165e2/volumes" Nov 01 22:27:51.172508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.172485 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:27:51.211954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.211929 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:27:51.311852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.311821 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\") pod \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " Nov 01 22:27:51.312050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.311949 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\" (UniqueName: \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\") pod \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\" (UID: \"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5\") " Nov 01 22:27:51.312157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.312139 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\") pod \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " Nov 01 22:27:51.312228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.312164 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume" pod "e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" (UID: "e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:51.312289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.312260 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\" (UniqueName: \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\") pod \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\" (UID: \"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4\") " Nov 01 22:27:51.312500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.312476 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume" pod "73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" (UID: "73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:51.313162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.313142 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:51.313162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.313162 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:51.314463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.314442 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh" pod "73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" (UID: "73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4"). InnerVolumeSpecName "kube-api-access-ncslh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:51.314724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.314706 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f" pod "e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" (UID: "e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5"). InnerVolumeSpecName "kube-api-access-44w9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:51.414451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.414420 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncslh\" (UniqueName: \"kubernetes.io/projected/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4-kube-api-access-ncslh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:51.414451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.414447 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44w9f\" (UniqueName: \"kubernetes.io/projected/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5-kube-api-access-44w9f\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:51.683956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.683919 4395 generic.go:356] "Generic (PLEG): container finished" podID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerID="fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761" exitCode=0 Nov 01 22:27:51.684190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.683963 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" Nov 01 22:27:51.684190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.684008 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" event={"ID":"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5","Type":"ContainerDied","Data":"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761"} Nov 01 22:27:51.684190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.684088 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn" event={"ID":"e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5","Type":"ContainerDied","Data":"bd8119d612de46fa56e4b32923b85578066a685a4d78852eb3145e7a34c03607"} Nov 01 22:27:51.684190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.684119 4395 scope.go:122] "RemoveContainer" containerID="fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761" Nov 01 22:27:51.685628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.685609 4395 generic.go:356] "Generic (PLEG): container finished" podID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerID="f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f" exitCode=0 Nov 01 22:27:51.685923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.685643 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" Nov 01 22:27:51.685923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.685702 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" event={"ID":"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4","Type":"ContainerDied","Data":"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f"} Nov 01 22:27:51.685923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.685756 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85" event={"ID":"73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4","Type":"ContainerDied","Data":"f7feb451a9fea816a733cd9c4e20fee41b242477996f5b1af07b264e532068a6"} Nov 01 22:27:51.706299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.700773 4395 scope.go:122] "RemoveContainer" containerID="fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761" Nov 01 22:27:51.722052 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:51.714250 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761\": container with ID starting with fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761 not found: ID does not exist" containerID="fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761" Nov 01 22:27:51.722052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.714302 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761"} err="failed to get container status \"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761\": rpc error: code = NotFound desc = could not find container \"fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761\": container with ID starting with fb53109001c86707c2a5321d5d9f8408dcaf7463dc5005d83ce7c4daa0189761 not found: ID does not exist" Nov 01 22:27:51.722052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.714332 4395 scope.go:122] "RemoveContainer" containerID="f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f" Nov 01 22:27:51.766829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.764995 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:27:51.768206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.768187 4395 scope.go:122] "RemoveContainer" containerID="f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f" Nov 01 22:27:51.770359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.770213 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-d6e25f-689b6f998c-mlz85"] Nov 01 22:27:51.770481 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:51.770424 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f\": container with ID starting with f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f not found: ID does not exist" containerID="f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f" Nov 01 22:27:51.770614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.770467 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f"} err="failed to get container status \"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f\": rpc error: code = NotFound desc = could not find container \"f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f\": container with ID starting with f90a29e95b60e391dbf8310736fec0151bf0bb0ecec4d30d4455efa984da743f not found: ID does not exist" Nov 01 22:27:51.776166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.776143 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:27:51.781389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:51.781366 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-d6e25f-dc644b8-wmqhn"] Nov 01 22:27:52.184622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.184584 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:27:52.185161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.184862 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" podUID="cb175887-0bf0-4c55-a1af-f5492f3a9506" containerName="maas-api" containerID="cri-o://23ace1ff377a2099a11588820cab0f57e8abd41642934b454a82996749968eb2" gracePeriod=30 Nov 01 22:27:52.308242 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:27:52.308205 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:27:52.698661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.698622 4395 generic.go:356] "Generic (PLEG): container finished" podID="cb175887-0bf0-4c55-a1af-f5492f3a9506" containerID="23ace1ff377a2099a11588820cab0f57e8abd41642934b454a82996749968eb2" exitCode=0 Nov 01 22:27:52.698829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.698671 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" event={"ID":"cb175887-0bf0-4c55-a1af-f5492f3a9506","Type":"ContainerDied","Data":"23ace1ff377a2099a11588820cab0f57e8abd41642934b454a82996749968eb2"} Nov 01 22:27:52.698829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.698746 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" event={"ID":"cb175887-0bf0-4c55-a1af-f5492f3a9506","Type":"ContainerDied","Data":"2ccb13a60b58828aebccf6893a570478567a505b0c698b03f744cace26273789"} Nov 01 22:27:52.698829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.698771 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ccb13a60b58828aebccf6893a570478567a505b0c698b03f744cace26273789" Nov 01 22:27:52.716516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.716443 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:27:52.888053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.869685 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\" (UniqueName: \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\") pod \"cb175887-0bf0-4c55-a1af-f5492f3a9506\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " Nov 01 22:27:52.888053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.875890 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" containerID="cri-o://cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465" gracePeriod=25 Nov 01 22:27:52.888053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.876985 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\") pod \"cb175887-0bf0-4c55-a1af-f5492f3a9506\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " Nov 01 22:27:52.888053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.877443 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca" pod "cb175887-0bf0-4c55-a1af-f5492f3a9506" (UID: "cb175887-0bf0-4c55-a1af-f5492f3a9506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:27:52.888053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.877546 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\") pod \"cb175887-0bf0-4c55-a1af-f5492f3a9506\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " Nov 01 22:27:52.894078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.890475 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" containerID="cri-o://f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418" gracePeriod=25 Nov 01 22:27:52.915052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.901759 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\") pod \"cb175887-0bf0-4c55-a1af-f5492f3a9506\" (UID: \"cb175887-0bf0-4c55-a1af-f5492f3a9506\") " Nov 01 22:27:52.915052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.902847 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb175887-0bf0-4c55-a1af-f5492f3a9506-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:52.915769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.915732 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs" pod "cb175887-0bf0-4c55-a1af-f5492f3a9506" (UID: "cb175887-0bf0-4c55-a1af-f5492f3a9506"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:52.919359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.919211 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls" pod "cb175887-0bf0-4c55-a1af-f5492f3a9506" (UID: "cb175887-0bf0-4c55-a1af-f5492f3a9506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:27:52.924349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:52.922261 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6" pod "cb175887-0bf0-4c55-a1af-f5492f3a9506" (UID: "cb175887-0bf0-4c55-a1af-f5492f3a9506"). InnerVolumeSpecName "kube-api-access-qjhh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:53.004478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.004449 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.004605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.004484 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhh6\" (UniqueName: \"kubernetes.io/projected/cb175887-0bf0-4c55-a1af-f5492f3a9506-kube-api-access-qjhh6\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.004605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.004502 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb175887-0bf0-4c55-a1af-f5492f3a9506-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:53.070901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.070867 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:27:53.071863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071844 4395 cpu_manager.go:414] "containerMap: removing container" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" Nov 01 22:27:53.071923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071866 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" Nov 01 22:27:53.071923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071883 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d57d79b3-a752-4808-9558-eef9087cd525" containerName="payload-pre-processing" Nov 01 22:27:53.071923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071889 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57d79b3-a752-4808-9558-eef9087cd525" containerName="payload-pre-processing" Nov 01 22:27:53.071923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071913 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cb175887-0bf0-4c55-a1af-f5492f3a9506" containerName="maas-api" Nov 01 22:27:53.071923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071918 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb175887-0bf0-4c55-a1af-f5492f3a9506" containerName="maas-api" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071939 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071943 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071956 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071960 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071967 4395 cpu_manager.go:414] "containerMap: removing container" podUID="865dd45e-fe43-439b-ab89-a44000605061" containerName="payload-pre-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.071972 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865dd45e-fe43-439b-ab89-a44000605061" containerName="payload-pre-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072086 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57d79b3-a752-4808-9558-eef9087cd525" containerName="payload-pre-processing" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072100 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb175887-0bf0-4c55-a1af-f5492f3a9506" containerName="maas-api" Nov 01 22:27:53.072108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072110 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba346d51-88f9-468a-817e-d5963a8165e2" containerName="payload-processing" Nov 01 22:27:53.072383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072117 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" containerName="payload-processing" Nov 01 22:27:53.072383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072124 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="865dd45e-fe43-439b-ab89-a44000605061" containerName="payload-pre-processing" Nov 01 22:27:53.072383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.072132 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" containerName="payload-pre-processing" Nov 01 22:27:53.076059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.076020 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.094590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.094564 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:27:53.099271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.099242 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:27:53.105415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105393 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105455 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105588 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105617 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105702 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105785 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105834 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105867 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqh4d\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.105945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.105931 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.106324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.106305 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.137003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.136970 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:27:53.140947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.140800 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:27:53.150412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.150390 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.155594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.155364 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:27:53.174875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.174848 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4" path="/var/lib/kubelet/pods/73d8dbb8-1d5b-4640-93f9-c1f2763ff2d4/volumes" Nov 01 22:27:53.175379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.175364 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5" path="/var/lib/kubelet/pods/e9538ac7-9b20-46f9-a6fc-d0fef2bbacc5/volumes" Nov 01 22:27:53.207971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.207942 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.207987 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.208388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208157 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.208388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208250 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208364 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208408 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208458 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208478 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208521 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wgs\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208557 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208641 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208677 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208710 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208791 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208817 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208817 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.208909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.208857 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.209167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209041 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.209167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209117 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpb8\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.209167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209131 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.209331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209169 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.209331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209207 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.209331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209250 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.209331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209285 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqh4d\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.209331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209311 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.209780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209368 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.209951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209919 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.210156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209956 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.210156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.209995 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.210156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.210063 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.210156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.210061 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.210547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.210504 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.211876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.211854 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.212492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.212464 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.221196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.221168 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.221279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.221263 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqh4d\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.311814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311727 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.311814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311768 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.311814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311798 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311828 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpb8\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311863 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311916 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311956 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.311996 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312043 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312108 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312140 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312172 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312216 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312330 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312335 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312422 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wgs\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312454 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312546 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312579 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312886 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.312989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312933 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.312989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312588 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.313196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.312991 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.313196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.313167 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.313389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.313365 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.313645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.313621 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.313960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.313937 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.313960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.313995 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.316776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.316727 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.317308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.317093 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.317308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.317279 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.317308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.317303 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.328210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.328186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wgs\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.328322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.328239 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.328897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.328876 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.329899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.329877 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpb8\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.395475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.395424 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:53.434371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.434341 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:53.473687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.473639 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:53.702359 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:27:53.702325 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7525e218_1750_4b26_8d94_6a2a2e7a84a2.slice/crio-19965f62c995347f72ba08dee6f303f88fd9fcdd2e11b6a615c68e45c9e3db34 WatchSource:0}: Error finding container 19965f62c995347f72ba08dee6f303f88fd9fcdd2e11b6a615c68e45c9e3db34: Status 404 returned error can't find the container with id 19965f62c995347f72ba08dee6f303f88fd9fcdd2e11b6a615c68e45c9e3db34 Nov 01 22:27:53.705683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.705656 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:27:53.707268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.707229 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.707380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.707317 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.707380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.707359 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.712128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.712107 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn" Nov 01 22:27:53.755198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.755169 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:27:53.759574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.759530 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-da34ca-7d4f6c94d7-dxztn"] Nov 01 22:27:53.974264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.974162 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:27:53.985319 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:27:53.983459 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b84cb7c_b176_4705_aa31_9f29cb0d650f.slice/crio-34fde81a07a764d3e0567b02cf6bc3299d294c412ec1383e6554f4707c588405 WatchSource:0}: Error finding container 34fde81a07a764d3e0567b02cf6bc3299d294c412ec1383e6554f4707c588405: Status 404 returned error can't find the container with id 34fde81a07a764d3e0567b02cf6bc3299d294c412ec1383e6554f4707c588405 Nov 01 22:27:53.987678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.987640 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.987810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.987780 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.987904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.987882 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.990755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.990016 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.990755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.990166 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.990755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.990235 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:27:53.999845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:53.999826 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:27:54.752449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.752411 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" event={"ID":"1b84cb7c-b176-4705-aa31-9f29cb0d650f","Type":"ContainerStarted","Data":"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4"} Nov 01 22:27:54.753129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.753102 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" event={"ID":"1b84cb7c-b176-4705-aa31-9f29cb0d650f","Type":"ContainerStarted","Data":"34fde81a07a764d3e0567b02cf6bc3299d294c412ec1383e6554f4707c588405"} Nov 01 22:27:54.766269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.766119 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" event={"ID":"8139486a-45ba-4f0b-91e7-d20c1deedd62","Type":"ContainerStarted","Data":"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0"} Nov 01 22:27:54.766620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.766594 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" event={"ID":"8139486a-45ba-4f0b-91e7-d20c1deedd62","Type":"ContainerStarted","Data":"c834c7e1e079f0e0ae6224dfcf9343a9176361c8c383fab6b14c94eae3a15ba0"} Nov 01 22:27:54.772345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.772316 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" event={"ID":"7525e218-1750-4b26-8d94-6a2a2e7a84a2","Type":"ContainerStarted","Data":"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151"} Nov 01 22:27:54.772499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.772367 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" event={"ID":"7525e218-1750-4b26-8d94-6a2a2e7a84a2","Type":"ContainerStarted","Data":"19965f62c995347f72ba08dee6f303f88fd9fcdd2e11b6a615c68e45c9e3db34"} Nov 01 22:27:54.821199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:54.814984 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podStartSLOduration=1.814965889 podStartE2EDuration="1.814965889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.784490557 +0000 UTC m=+2621.758393096" watchObservedRunningTime="2026-11-01 22:27:54.814965889 +0000 UTC m=+2621.788868429" Nov 01 22:27:55.116421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.116303 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podStartSLOduration=2.116279526 podStartE2EDuration="2.116279526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.8993922 +0000 UTC m=+2621.873294752" watchObservedRunningTime="2026-11-01 22:27:55.116279526 +0000 UTC m=+2622.090182065" Nov 01 22:27:55.187052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.148947 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podStartSLOduration=2.148927334 podStartE2EDuration="2.148927334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.947830371 +0000 UTC m=+2621.921732912" watchObservedRunningTime="2026-11-01 22:27:55.148927334 +0000 UTC m=+2622.122829874" Nov 01 22:27:55.488109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.437416 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:27:55.488109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.437481 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:27:55.504094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.493769 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:27:55.504094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.493827 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:27:55.528095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.511959 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb175887-0bf0-4c55-a1af-f5492f3a9506" path="/var/lib/kubelet/pods/cb175887-0bf0-4c55-a1af-f5492f3a9506/volumes" Nov 01 22:27:55.591360 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.550651 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:27:55.591687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.591660 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:27:55.591839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.591828 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:27:55.601054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.533313 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:27:55.605052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:55.601309 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:27:56.398418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.398383 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:27:56.399014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.398981 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:27:56.440197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.440150 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:27:56.440451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.440428 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:27:56.543266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.543230 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:27:56.543869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:56.543842 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:27:57.397794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.397757 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:27:57.398111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.398084 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:27:57.438952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.438919 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:27:57.439529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.439503 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:27:57.478882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.478423 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:27:57.478882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:57.478484 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:27:58.265053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.264942 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:58.267187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.267148 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:27:58.433052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.432545 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:27:58.433052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.432607 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:27:58.437051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.436384 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:27:58.437051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.436435 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:27:58.486055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.484388 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:27:58.486055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:58.484451 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:27:59.473614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.468148 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:27:59.473614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.468201 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:27:59.473614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.468542 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:27:59.473614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.468567 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:27:59.551999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.551961 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:27:59.552582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:27:59.552550 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:28:00.263135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.263097 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:28:00.265579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.265547 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" containerID="cri-o://888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284" gracePeriod=30 Nov 01 22:28:00.415053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.400434 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:28:00.415053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.400504 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:28:00.476911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.476874 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:28:00.555123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.550104 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:28:00.555123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.510778 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:28:00.555123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:00.550553 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:28:01.223053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.222109 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:01.245463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.241241 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:01.245463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.241362 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.247021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.246979 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db715b-dockercfg-mlmn9\"" Nov 01 22:28:01.262651 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.262624 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db715b\"" Nov 01 22:28:01.370070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.357242 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.370070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.360204 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4msw\" (UniqueName: \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.404092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.398272 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:28:01.404092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.398350 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:28:01.440053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.438845 4395 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:28:01.440053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.438921 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Nov 01 22:28:01.463268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.463232 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.463559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.463540 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4msw\" (UniqueName: \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.475142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.464904 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.476467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.476441 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:28:01.476641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.476620 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Nov 01 22:28:01.515871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.515834 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4msw\" (UniqueName: \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\") pod \"payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:01.645054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:01.632503 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:02.277131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.271312 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:02.277131 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:02.274087 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6cd6c6_68d8_4ce7_81da_50e5c0b9a358.slice/crio-4e59e684cfcdc3ef6efce03a257b3e657beab66cc9b8c335f1c5f28295ad90ec WatchSource:0}: Error finding container 4e59e684cfcdc3ef6efce03a257b3e657beab66cc9b8c335f1c5f28295ad90ec: Status 404 returned error can't find the container with id 4e59e684cfcdc3ef6efce03a257b3e657beab66cc9b8c335f1c5f28295ad90ec Nov 01 22:28:02.410165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.408015 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:28:02.410165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.409727 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:28:02.411130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.411104 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:28:02.484079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.481204 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:28:02.484079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.482542 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:28:02.487226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.486335 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:28:02.488404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.487431 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:28:02.488404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.487595 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:28:02.499587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.498321 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:28:02.605498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.602373 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" event={"ID":"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358","Type":"ContainerStarted","Data":"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e"} Nov 01 22:28:02.605498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.602449 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" event={"ID":"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358","Type":"ContainerStarted","Data":"4e59e684cfcdc3ef6efce03a257b3e657beab66cc9b8c335f1c5f28295ad90ec"} Nov 01 22:28:02.605498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.605256 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:02.628417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:02.628358 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" podStartSLOduration=1.628339864 podStartE2EDuration="1.628339864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:02.625565857 +0000 UTC m=+2629.599468395" watchObservedRunningTime="2026-11-01 22:28:02.628339864 +0000 UTC m=+2629.602242404" Nov 01 22:28:05.569233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.569199 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:28:05.578423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578382 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.578561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578489 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.578643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578605 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578791 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578854 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578901 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578940 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.578982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.579075 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\") pod \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\" (UID: \"62454f8a-f535-4e55-93ae-a5f004f0d1fe\") " Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.579075 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:05.579765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.579591 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:05.580312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.579872 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.580312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.579894 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.580312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.580088 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:05.580575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.580539 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:05.581844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.581820 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:05.582016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.581997 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:05.584058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.584005 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:05.590558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.590530 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:28:05.591811 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.591782 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d" pod "62454f8a-f535-4e55-93ae-a5f004f0d1fe" (UID: "62454f8a-f535-4e55-93ae-a5f004f0d1fe"). InnerVolumeSpecName "kube-api-access-rqw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:05.630967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.630937 4395 generic.go:356] "Generic (PLEG): container finished" podID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerID="888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284" exitCode=0 Nov 01 22:28:05.631129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.631109 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerDied","Data":"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284"} Nov 01 22:28:05.631199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.631171 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" event={"ID":"62454f8a-f535-4e55-93ae-a5f004f0d1fe","Type":"ContainerDied","Data":"1c0a0d647c15373e06459f5c685b72bcde32e23a398522ceb55a8050134129fe"} Nov 01 22:28:05.631245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.631202 4395 scope.go:122] "RemoveContainer" containerID="888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284" Nov 01 22:28:05.631383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.631367 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c" Nov 01 22:28:05.647110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.647089 4395 scope.go:122] "RemoveContainer" containerID="6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad" Nov 01 22:28:05.669163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.669143 4395 scope.go:122] "RemoveContainer" containerID="888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284" Nov 01 22:28:05.669245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.669237 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:28:05.669454 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:05.669431 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284\": container with ID starting with 888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284 not found: ID does not exist" containerID="888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284" Nov 01 22:28:05.669570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.669460 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284"} err="failed to get container status \"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284\": rpc error: code = NotFound desc = could not find container \"888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284\": container with ID starting with 888dc5310ef140db0da2598f2ae8c0061691dacdbe15b0c369958eaf2a191284 not found: ID does not exist" Nov 01 22:28:05.669570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.669478 4395 scope.go:122] "RemoveContainer" containerID="6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad" Nov 01 22:28:05.669739 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:05.669722 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad\": container with ID starting with 6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad not found: ID does not exist" containerID="6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad" Nov 01 22:28:05.669802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.669744 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad"} err="failed to get container status \"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad\": rpc error: code = NotFound desc = could not find container \"6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad\": container with ID starting with 6e97575e4cd83fcc0827b02135166218b35cc5de355873dac04a1fa580601cad not found: ID does not exist" Nov 01 22:28:05.672342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.672318 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-g9k8c"] Nov 01 22:28:05.679964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.679939 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 22:28:05.681182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681151 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqw2d\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-kube-api-access-rqw2d\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681178 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681192 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681205 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681214 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681223 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.681311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.681231 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62454f8a-f535-4e55-93ae-a5f004f0d1fe-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:05.824634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.824545 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:28:05.824813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.824792 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" podUID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" containerName="payload-processing" containerID="cri-o://430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2" gracePeriod=30 Nov 01 22:28:05.830270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.830245 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:28:05.830537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:05.830506 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" podUID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" containerName="payload-pre-processing" containerID="cri-o://0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5" gracePeriod=30 Nov 01 22:28:06.405560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.405535 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:28:06.509096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.485045 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:28:06.517827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.517464 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\") pod \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " Nov 01 22:28:06.517827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.517531 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\" (UniqueName: \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\") pod \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\" (UID: \"a419fbeb-e8d7-4253-8451-ae4ec39cda6a\") " Nov 01 22:28:06.520934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.519284 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume" pod "a419fbeb-e8d7-4253-8451-ae4ec39cda6a" (UID: "a419fbeb-e8d7-4253-8451-ae4ec39cda6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:06.525114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.525081 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx" pod "a419fbeb-e8d7-4253-8451-ae4ec39cda6a" (UID: "a419fbeb-e8d7-4253-8451-ae4ec39cda6a"). InnerVolumeSpecName "kube-api-access-wzwdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:06.619158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619123 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\" (UniqueName: \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\") pod \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " Nov 01 22:28:06.619638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619186 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\") pod \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\" (UID: \"3f23c08b-0908-45c9-9b98-4f3cf30dc371\") " Nov 01 22:28:06.619638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619544 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume" pod "3f23c08b-0908-45c9-9b98-4f3cf30dc371" (UID: "3f23c08b-0908-45c9-9b98-4f3cf30dc371"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:06.619792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619774 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f23c08b-0908-45c9-9b98-4f3cf30dc371-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.619832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619797 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.619832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.619807 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwdx\" (UniqueName: \"kubernetes.io/projected/a419fbeb-e8d7-4253-8451-ae4ec39cda6a-kube-api-access-wzwdx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:06.621571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.621543 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49" pod "3f23c08b-0908-45c9-9b98-4f3cf30dc371" (UID: "3f23c08b-0908-45c9-9b98-4f3cf30dc371"). InnerVolumeSpecName "kube-api-access-wld49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:06.644412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.644378 4395 generic.go:356] "Generic (PLEG): container finished" podID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" containerID="0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5" exitCode=0 Nov 01 22:28:06.644554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.644426 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" Nov 01 22:28:06.644554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.644453 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" event={"ID":"a419fbeb-e8d7-4253-8451-ae4ec39cda6a","Type":"ContainerDied","Data":"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5"} Nov 01 22:28:06.644554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.644513 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c" event={"ID":"a419fbeb-e8d7-4253-8451-ae4ec39cda6a","Type":"ContainerDied","Data":"3cd7bd7b99811264c3705a940bda7fb0a65bf77e8a4fb612c5e7c9dbc695934a"} Nov 01 22:28:06.644554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.644544 4395 scope.go:122] "RemoveContainer" containerID="0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5" Nov 01 22:28:06.646323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.646296 4395 generic.go:356] "Generic (PLEG): container finished" podID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" containerID="430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2" exitCode=0 Nov 01 22:28:06.646477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.646372 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" Nov 01 22:28:06.646823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.646374 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" event={"ID":"3f23c08b-0908-45c9-9b98-4f3cf30dc371","Type":"ContainerDied","Data":"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2"} Nov 01 22:28:06.646921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.646868 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf" event={"ID":"3f23c08b-0908-45c9-9b98-4f3cf30dc371","Type":"ContainerDied","Data":"23ddd8b31997d102dd8656159c0db2caa663bfebfa642153c85754c71408b6bc"} Nov 01 22:28:06.659683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.659666 4395 scope.go:122] "RemoveContainer" containerID="0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5" Nov 01 22:28:06.659944 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:06.659926 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5\": container with ID starting with 0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5 not found: ID does not exist" containerID="0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5" Nov 01 22:28:06.659992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.659955 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5"} err="failed to get container status \"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5\": rpc error: code = NotFound desc = could not find container \"0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5\": container with ID starting with 0ae44f80114a87940c976720b2a3b468b2d53c2af843bb27a84b3def4adfd1f5 not found: ID does not exist" Nov 01 22:28:06.659992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.659971 4395 scope.go:122] "RemoveContainer" containerID="430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2" Nov 01 22:28:06.669327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.669310 4395 scope.go:122] "RemoveContainer" containerID="430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2" Nov 01 22:28:06.669568 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:06.669545 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2\": container with ID starting with 430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2 not found: ID does not exist" containerID="430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2" Nov 01 22:28:06.669621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.669576 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2"} err="failed to get container status \"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2\": rpc error: code = NotFound desc = could not find container \"430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2\": container with ID starting with 430107c07fea1771d40bbbd54a373f92bbbd5e576f304fba757c782357784ff2 not found: ID does not exist" Nov 01 22:28:06.680276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.680253 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:28:06.684469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.684449 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-6446dc7b59-bpd2c"] Nov 01 22:28:06.696255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.696231 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:28:06.703790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.703765 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-76979cf7dd-vvvpf"] Nov 01 22:28:06.721321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:06.721293 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wld49\" (UniqueName: \"kubernetes.io/projected/3f23c08b-0908-45c9-9b98-4f3cf30dc371-kube-api-access-wld49\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:07.166809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:07.166776 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" path="/var/lib/kubelet/pods/3f23c08b-0908-45c9-9b98-4f3cf30dc371/volumes" Nov 01 22:28:07.167326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:07.167311 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" path="/var/lib/kubelet/pods/62454f8a-f535-4e55-93ae-a5f004f0d1fe/volumes" Nov 01 22:28:07.167768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:07.167746 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" path="/var/lib/kubelet/pods/a419fbeb-e8d7-4253-8451-ae4ec39cda6a/volumes" Nov 01 22:28:08.248965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:08.248926 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:08.249429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:08.248938 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:08.930298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:08.930200 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:28:08.930531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:08.930504 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" podUID="7cba018a-1697-41fd-9acd-46ce936433c3" containerName="maas-api" containerID="cri-o://956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0" gracePeriod=30 Nov 01 22:28:09.525371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.525345 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:28:09.534468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.534445 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\" (UniqueName: \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\") pod \"7cba018a-1697-41fd-9acd-46ce936433c3\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " Nov 01 22:28:09.534628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.534605 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\") pod \"7cba018a-1697-41fd-9acd-46ce936433c3\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " Nov 01 22:28:09.534748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.534728 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\") pod \"7cba018a-1697-41fd-9acd-46ce936433c3\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " Nov 01 22:28:09.534879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.534826 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\") pod \"7cba018a-1697-41fd-9acd-46ce936433c3\" (UID: \"7cba018a-1697-41fd-9acd-46ce936433c3\") " Nov 01 22:28:09.535152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.535123 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca" pod "7cba018a-1697-41fd-9acd-46ce936433c3" (UID: "7cba018a-1697-41fd-9acd-46ce936433c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:09.535855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.535827 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cba018a-1697-41fd-9acd-46ce936433c3-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:09.537161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.537135 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc" pod "7cba018a-1697-41fd-9acd-46ce936433c3" (UID: "7cba018a-1697-41fd-9acd-46ce936433c3"). InnerVolumeSpecName "kube-api-access-ng2cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:09.537513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.537490 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls" pod "7cba018a-1697-41fd-9acd-46ce936433c3" (UID: "7cba018a-1697-41fd-9acd-46ce936433c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:09.537739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.537717 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs" pod "7cba018a-1697-41fd-9acd-46ce936433c3" (UID: "7cba018a-1697-41fd-9acd-46ce936433c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:09.637321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.637249 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:09.637321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.637276 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cba018a-1697-41fd-9acd-46ce936433c3-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:09.637321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.637288 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2cc\" (UniqueName: \"kubernetes.io/projected/7cba018a-1697-41fd-9acd-46ce936433c3-kube-api-access-ng2cc\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:09.730064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.726379 4395 generic.go:356] "Generic (PLEG): container finished" podID="7cba018a-1697-41fd-9acd-46ce936433c3" containerID="956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0" exitCode=0 Nov 01 22:28:09.730064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.726517 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" event={"ID":"7cba018a-1697-41fd-9acd-46ce936433c3","Type":"ContainerDied","Data":"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0"} Nov 01 22:28:09.730064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.726560 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" event={"ID":"7cba018a-1697-41fd-9acd-46ce936433c3","Type":"ContainerDied","Data":"a24d9017c24c86a32cb7d88abbaba901f05be95d77d10a526721972b7798a57b"} Nov 01 22:28:09.730064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.726588 4395 scope.go:122] "RemoveContainer" containerID="956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0" Nov 01 22:28:09.730064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.726728 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9" Nov 01 22:28:09.746008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.745986 4395 scope.go:122] "RemoveContainer" containerID="956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0" Nov 01 22:28:09.746291 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:09.746263 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0\": container with ID starting with 956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0 not found: ID does not exist" containerID="956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0" Nov 01 22:28:09.746330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.746305 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0"} err="failed to get container status \"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0\": rpc error: code = NotFound desc = could not find container \"956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0\": container with ID starting with 956082513ed35e0f9deb8bcdb13ad54776df0bc449bc8ac4cec295dde6cac4b0 not found: ID does not exist" Nov 01 22:28:09.756104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.756082 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:28:09.760048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:09.760015 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-de0cba-65486884f6-smfz9"] Nov 01 22:28:11.130570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:11.130532 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 22:28:11.175335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:11.175302 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cba018a-1697-41fd-9acd-46ce936433c3" path="/var/lib/kubelet/pods/7cba018a-1697-41fd-9acd-46ce936433c3/volumes" Nov 01 22:28:13.211364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.211335 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:28:13.283862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.283837 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:28:13.348115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.342857 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\" (UniqueName: \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\") pod \"09f491e2-7944-449a-bc2a-25d212be6c57\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " Nov 01 22:28:13.348115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.343256 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\") pod \"09f491e2-7944-449a-bc2a-25d212be6c57\" (UID: \"09f491e2-7944-449a-bc2a-25d212be6c57\") " Nov 01 22:28:13.356669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.356619 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume" pod "09f491e2-7944-449a-bc2a-25d212be6c57" (UID: "09f491e2-7944-449a-bc2a-25d212be6c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:13.357313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.357212 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5" pod "09f491e2-7944-449a-bc2a-25d212be6c57" (UID: "09f491e2-7944-449a-bc2a-25d212be6c57"). InnerVolumeSpecName "kube-api-access-jlwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:13.444535 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.444445 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\") pod \"e63208df-f792-41e8-9f8a-cd459084c2d6\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " Nov 01 22:28:13.444691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.444566 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\" (UniqueName: \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\") pod \"e63208df-f792-41e8-9f8a-cd459084c2d6\" (UID: \"e63208df-f792-41e8-9f8a-cd459084c2d6\") " Nov 01 22:28:13.444864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.444835 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume" pod "e63208df-f792-41e8-9f8a-cd459084c2d6" (UID: "e63208df-f792-41e8-9f8a-cd459084c2d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:13.445182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.445162 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63208df-f792-41e8-9f8a-cd459084c2d6-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:13.445286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.445185 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f491e2-7944-449a-bc2a-25d212be6c57-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:13.445286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.445199 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwx5\" (UniqueName: \"kubernetes.io/projected/09f491e2-7944-449a-bc2a-25d212be6c57-kube-api-access-jlwx5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:13.446912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.446887 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs" pod "e63208df-f792-41e8-9f8a-cd459084c2d6" (UID: "e63208df-f792-41e8-9f8a-cd459084c2d6"). InnerVolumeSpecName "kube-api-access-g9vxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:13.546990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.546953 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vxs\" (UniqueName: \"kubernetes.io/projected/e63208df-f792-41e8-9f8a-cd459084c2d6-kube-api-access-g9vxs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:13.799055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.798809 4395 generic.go:356] "Generic (PLEG): container finished" podID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerID="f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418" exitCode=0 Nov 01 22:28:13.799055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.798931 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" event={"ID":"e63208df-f792-41e8-9f8a-cd459084c2d6","Type":"ContainerDied","Data":"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418"} Nov 01 22:28:13.799055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.798991 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" event={"ID":"e63208df-f792-41e8-9f8a-cd459084c2d6","Type":"ContainerDied","Data":"1668e9f5a2edb798d7b406830686028812765254d11b579513b95ac8b0792b83"} Nov 01 22:28:13.799055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.799046 4395 scope.go:122] "RemoveContainer" containerID="f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418" Nov 01 22:28:13.799441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.799234 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5" Nov 01 22:28:13.844373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.844183 4395 generic.go:356] "Generic (PLEG): container finished" podID="09f491e2-7944-449a-bc2a-25d212be6c57" containerID="cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465" exitCode=0 Nov 01 22:28:13.844373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.844263 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" Nov 01 22:28:13.844554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.844457 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" event={"ID":"09f491e2-7944-449a-bc2a-25d212be6c57","Type":"ContainerDied","Data":"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465"} Nov 01 22:28:13.844554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.844506 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8" event={"ID":"09f491e2-7944-449a-bc2a-25d212be6c57","Type":"ContainerDied","Data":"36a91ca7b854ee2c99bbeebdb3be5d77435873b0db6b5e0308df650e4154be00"} Nov 01 22:28:13.861674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.861627 4395 scope.go:122] "RemoveContainer" containerID="f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418" Nov 01 22:28:13.868190 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:13.866790 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418\": container with ID starting with f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418 not found: ID does not exist" containerID="f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418" Nov 01 22:28:13.868190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.866843 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418"} err="failed to get container status \"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418\": rpc error: code = NotFound desc = could not find container \"f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418\": container with ID starting with f034cc74c99c054f0b717a04066ee9b9ec95026e60052664d8696b75a4ff8418 not found: ID does not exist" Nov 01 22:28:13.868190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.866913 4395 scope.go:122] "RemoveContainer" containerID="cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465" Nov 01 22:28:13.879817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.878104 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:28:13.885514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.884180 4395 scope.go:122] "RemoveContainer" containerID="cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465" Nov 01 22:28:13.885514 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:13.884532 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465\": container with ID starting with cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465 not found: ID does not exist" containerID="cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465" Nov 01 22:28:13.885514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.884559 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465"} err="failed to get container status \"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465\": rpc error: code = NotFound desc = could not find container \"cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465\": container with ID starting with cfe5d857d8754f0f3e86a78e979b2fa5257d55e9b6a260e2f2339678e10f2465 not found: ID does not exist" Nov 01 22:28:13.887127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.886400 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-da34ca-c997c8c67-7zfp5"] Nov 01 22:28:13.898862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.898834 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:28:13.905151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:13.905129 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-da34ca-5d7fb98df4-m5gv8"] Nov 01 22:28:14.070893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:14.070809 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:28:14.623133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:14.623098 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:15.169542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:15.169513 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" path="/var/lib/kubelet/pods/09f491e2-7944-449a-bc2a-25d212be6c57/volumes" Nov 01 22:28:15.169974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:15.169959 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" path="/var/lib/kubelet/pods/e63208df-f792-41e8-9f8a-cd459084c2d6/volumes" Nov 01 22:28:16.131314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:16.131248 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" containerID="cri-o://f5da4ed42ecd616cff0c36af64d07f517d4ccdd9ac728cc443e045b3c2a67860" gracePeriod=25 Nov 01 22:28:16.409681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:16.409600 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:28:16.410162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:16.410123 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" containerID="cri-o://3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569" gracePeriod=30 Nov 01 22:28:16.769152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:16.769121 4395 scope.go:122] "RemoveContainer" containerID="23ace1ff377a2099a11588820cab0f57e8abd41642934b454a82996749968eb2" Nov 01 22:28:19.541519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.541480 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:28:19.541946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.541533 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:28:19.761907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.761863 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:28:19.762988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.762956 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7cba018a-1697-41fd-9acd-46ce936433c3" containerName="maas-api" Nov 01 22:28:19.762988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.762987 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cba018a-1697-41fd-9acd-46ce936433c3" containerName="maas-api" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763042 4395 cpu_manager.go:414] "containerMap: removing container" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763051 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763064 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" containerName="payload-pre-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763073 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" containerName="payload-pre-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763087 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763095 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763161 4395 cpu_manager.go:414] "containerMap: removing container" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763168 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763183 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" containerName="payload-processing" Nov 01 22:28:19.763295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763190 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" containerName="payload-processing" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763359 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f491e2-7944-449a-bc2a-25d212be6c57" containerName="payload-processing" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763378 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f23c08b-0908-45c9-9b98-4f3cf30dc371" containerName="payload-processing" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763391 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cba018a-1697-41fd-9acd-46ce936433c3" containerName="maas-api" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763405 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763414 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a419fbeb-e8d7-4253-8451-ae4ec39cda6a" containerName="payload-pre-processing" Nov 01 22:28:19.763847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.763425 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63208df-f792-41e8-9f8a-cd459084c2d6" containerName="payload-pre-processing" Nov 01 22:28:19.779703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.772087 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.779703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.779697 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:28:19.785315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.785272 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-db715b-serving-cert\"" Nov 01 22:28:19.893362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.893255 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq26s\" (UniqueName: \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.893362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.893316 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.893591 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.893551 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.893646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.893612 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.997062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.996569 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.997062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.996677 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.997062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.997009 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq26s\" (UniqueName: \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.997399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.997089 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:19.998453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:19.998421 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:20.004720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.004691 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:20.005244 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.005218 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:20.011712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.011688 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq26s\" (UniqueName: \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\") pod \"maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:20.093637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.093616 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:20.264639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.264595 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:28:20.464402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.453381 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:28:20.507068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.494546 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:28:20.706183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.706128 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" containerID="cri-o://2576d0f5066dcdc772ac265f2cec109cf7038901aa4944592a4211a2befe3c04" gracePeriod=45 Nov 01 22:28:20.733566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:20.733528 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Nov 01 22:28:21.082399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.075837 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" event={"ID":"2eb969be-cd66-4fc3-b0c9-4f71025b06ff","Type":"ContainerStarted","Data":"f0729ba6e63cefb6ae70549aac30c49713d7502532158a81ff09e6dac60793e4"} Nov 01 22:28:21.082399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.077876 4395 generic.go:356] "Generic (PLEG): container finished" podID="15349852-76c5-49b2-9a02-385f2d225f17" containerID="2576d0f5066dcdc772ac265f2cec109cf7038901aa4944592a4211a2befe3c04" exitCode=0 Nov 01 22:28:21.082399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.078010 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" event={"ID":"15349852-76c5-49b2-9a02-385f2d225f17","Type":"ContainerDied","Data":"2576d0f5066dcdc772ac265f2cec109cf7038901aa4944592a4211a2befe3c04"} Nov 01 22:28:21.120144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.120118 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:28:21.191915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.191881 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.192106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.191998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.192269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192246 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.192341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192259 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.192341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192315 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.192470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192353 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\" (UniqueName: \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.192532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192517 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\" (UniqueName: \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.192586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.192556 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\") pod \"15349852-76c5-49b2-9a02-385f2d225f17\" (UID: \"15349852-76c5-49b2-9a02-385f2d225f17\") " Nov 01 22:28:21.195183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.193341 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.195183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.193365 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.199801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.199674 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.202099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.202018 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.234112 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.234776 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235519 4395 cpu_manager.go:414] "containerMap: removing container" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235536 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235575 4395 cpu_manager.go:414] "containerMap: removing container" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235584 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235750 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="15349852-76c5-49b2-9a02-385f2d225f17" containerName="main" Nov 01 22:28:21.236512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.235768 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="62454f8a-f535-4e55-93ae-a5f004f0d1fe" containerName="istio-proxy" Nov 01 22:28:21.236998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.236802 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s" pod "15349852-76c5-49b2-9a02-385f2d225f17" (UID: "15349852-76c5-49b2-9a02-385f2d225f17"). InnerVolumeSpecName "kube-api-access-zmx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:21.242408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.242260 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:28:21.242408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.242376 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.251078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.248241 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c6bb56-openshift-default-dockercfg-55rms\"" Nov 01 22:28:21.295650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295471 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295522 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295595 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295649 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295690 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295730 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295829 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295871 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbsh\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.295953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.295904 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.296287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.296089 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmx5s\" (UniqueName: \"kubernetes.io/projected/15349852-76c5-49b2-9a02-385f2d225f17-kube-api-access-zmx5s\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.296287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.296108 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.296287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.296122 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15349852-76c5-49b2-9a02-385f2d225f17-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.296287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.296136 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15349852-76c5-49b2-9a02-385f2d225f17-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.426996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.417013 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.438309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.438277 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.438596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.438578 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.438754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.438737 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.438894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.438882 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.439138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.439125 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.447075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.439262 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbsh\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.447350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.447318 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.448451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.448427 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.448732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.430792 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.449120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.449100 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.449926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.449827 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.450652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.447803 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.451294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.451270 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.537673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.537599 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.538160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.538136 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.550973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.550937 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.566049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.563695 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbsh\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\") pod \"e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.585794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.585183 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:21.839149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.839119 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:28:21.849361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.849338 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 22:28:21.904813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.904758 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:28:21.906150 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:21.906118 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2460d1_cb80_4a1a_9904_0eb21984fdc0.slice/crio-d4a2ae5e1ae5524b25dd1d0129016e74a58be1632667d6dfb6ddbf6bc3f2b897 WatchSource:0}: Error finding container d4a2ae5e1ae5524b25dd1d0129016e74a58be1632667d6dfb6ddbf6bc3f2b897: Status 404 returned error can't find the container with id d4a2ae5e1ae5524b25dd1d0129016e74a58be1632667d6dfb6ddbf6bc3f2b897 Nov 01 22:28:21.909304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.909273 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:28:21.909430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.909342 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:28:21.909430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.909372 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:28:21.959720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959692 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.959840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959767 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.959840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959803 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.959979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959868 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.959979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959965 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.960117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.959971 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.960170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960145 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.960242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960212 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.960313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960237 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:21.960313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960245 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.960313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960292 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.960491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960322 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\") pod \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\" (UID: \"835742f3-7833-4f1d-8bfd-8074f32a6b3c\") " Nov 01 22:28:21.960491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960354 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.960608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960578 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:21.960895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.960871 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.961112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.961075 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.961280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.961263 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.961403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.961388 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.961487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.961471 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:21.962748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.962708 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "kube-api-access-kmb9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:21.962867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.962819 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:28:21.963344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.963317 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:21.963569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:21.963548 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy" pod "835742f3-7833-4f1d-8bfd-8074f32a6b3c" (UID: "835742f3-7833-4f1d-8bfd-8074f32a6b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:22.062930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.062893 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.062930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.062929 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.063124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.062942 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.063124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.062955 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmb9g\" (UniqueName: \"kubernetes.io/projected/835742f3-7833-4f1d-8bfd-8074f32a6b3c-kube-api-access-kmb9g\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:22.131373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.131271 4395 generic.go:356] "Generic (PLEG): container finished" podID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerID="3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569" exitCode=0 Nov 01 22:28:22.131373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.131343 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" Nov 01 22:28:22.131373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.131357 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerDied","Data":"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569"} Nov 01 22:28:22.131660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.131423 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5" event={"ID":"835742f3-7833-4f1d-8bfd-8074f32a6b3c","Type":"ContainerDied","Data":"64ca058d67c9089f88d474ff029f39c39fa99af20e0769b90f30bcfeca50c85d"} Nov 01 22:28:22.131660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.131454 4395 scope.go:122] "RemoveContainer" containerID="3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569" Nov 01 22:28:22.134122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.134098 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" event={"ID":"2eb969be-cd66-4fc3-b0c9-4f71025b06ff","Type":"ContainerStarted","Data":"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9"} Nov 01 22:28:22.134277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.134260 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:22.147049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.142988 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" Nov 01 22:28:22.147049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.143100 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6" event={"ID":"15349852-76c5-49b2-9a02-385f2d225f17","Type":"ContainerDied","Data":"deaa2ef44f060a1ed52d313d7fa11f74423cd317b6bec609ebeb771bdf4047ce"} Nov 01 22:28:22.147049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.145792 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerStarted","Data":"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5"} Nov 01 22:28:22.147049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.145826 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerStarted","Data":"d4a2ae5e1ae5524b25dd1d0129016e74a58be1632667d6dfb6ddbf6bc3f2b897"} Nov 01 22:28:22.153157 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.152867 4395 scope.go:122] "RemoveContainer" containerID="10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc" Nov 01 22:28:22.159873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.159833 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" podStartSLOduration=2.451413595 podStartE2EDuration="3.159821024s" totalImagesPullingTime="708.407429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:22.157567998 +0000 UTC m=+2649.131470530" watchObservedRunningTime="2026-11-01 22:28:22.159821024 +0000 UTC m=+2649.133723563" Nov 01 22:28:22.172135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.171963 4395 scope.go:122] "RemoveContainer" containerID="3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569" Nov 01 22:28:22.173213 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:22.173184 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569\": container with ID starting with 3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569 not found: ID does not exist" containerID="3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569" Nov 01 22:28:22.173352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.173226 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569"} err="failed to get container status \"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569\": rpc error: code = NotFound desc = could not find container \"3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569\": container with ID starting with 3abf69aa917773002651e6745dfadc205876d0ca1f4abe7c8a364f8f397be569 not found: ID does not exist" Nov 01 22:28:22.173352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.173259 4395 scope.go:122] "RemoveContainer" containerID="10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc" Nov 01 22:28:22.173732 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:22.173582 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc\": container with ID starting with 10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc not found: ID does not exist" containerID="10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc" Nov 01 22:28:22.173732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.173619 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc"} err="failed to get container status \"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc\": rpc error: code = NotFound desc = could not find container \"10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc\": container with ID starting with 10555f07fdb314dc85a6e79a99a62701faeecacf8684f10611d1e1c8fc2e07cc not found: ID does not exist" Nov 01 22:28:22.173732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.173645 4395 scope.go:122] "RemoveContainer" containerID="2576d0f5066dcdc772ac265f2cec109cf7038901aa4944592a4211a2befe3c04" Nov 01 22:28:22.186052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.183387 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 22:28:22.188362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.188303 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-2-w6-fb9597-kserve-6f94988978-zmgc6"] Nov 01 22:28:22.203625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.203562 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:28:22.207946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.207924 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-de0cba-openshift-default-7788b4bcd7-whmj5"] Nov 01 22:28:22.227729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.227674 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podStartSLOduration=1.2276568939999999 podStartE2EDuration="1.227656894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:22.224149348 +0000 UTC m=+2649.198051887" watchObservedRunningTime="2026-11-01 22:28:22.227656894 +0000 UTC m=+2649.201559434" Nov 01 22:28:22.308720 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:22.308661 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:28:22.585667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.585630 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:22.592817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:22.592788 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:23.165670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:23.156743 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:23.171973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:23.171941 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15349852-76c5-49b2-9a02-385f2d225f17" path="/var/lib/kubelet/pods/15349852-76c5-49b2-9a02-385f2d225f17/volumes" Nov 01 22:28:23.172998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:23.172976 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" path="/var/lib/kubelet/pods/835742f3-7833-4f1d-8bfd-8074f32a6b3c/volumes" Nov 01 22:28:23.173904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:23.173866 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:28:24.684897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:24.684807 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:25.783642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.783609 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:28:25.784332 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.784317 4395 cpu_manager.go:414] "containerMap: removing container" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.784407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.784333 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.784484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.784475 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.784518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.784489 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.789856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.789837 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:25.795041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.795004 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6bb56\"" Nov 01 22:28:25.796625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.796601 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6bb56-dockercfg-6wx42\"" Nov 01 22:28:25.797962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.797925 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:28:25.833093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.833054 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:28:25.833254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.833126 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:25.833254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.833230 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pff\" (UniqueName: \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:25.835505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.835475 4395 cpu_manager.go:414] "containerMap: removing container" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.835505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.835506 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835742f3-7833-4f1d-8bfd-8074f32a6b3c" containerName="istio-proxy" Nov 01 22:28:25.845541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.845508 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:28:25.845899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.845831 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:25.967085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.942183 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrmw\" (UniqueName: \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:25.967085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.942378 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:25.967085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.942470 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pff\" (UniqueName: \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:25.967085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.942540 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:25.967085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:25.943773 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:26.126339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.121815 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pff\" (UniqueName: \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:26.131730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.131691 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrmw\" (UniqueName: \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:26.132002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.131983 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:26.133085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.133060 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:26.133350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.133324 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:26.148848 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.148826 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrmw\" (UniqueName: \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\") pod \"payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:26.400001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.399928 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:26.423150 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:26.423119 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382b2917_3446_498f_97d9_0a56ae83cd24.slice/crio-972da64675e22b32467e59bbdb0074470da0dc844ebb3000cdabeac89ed179a1 WatchSource:0}: Error finding container 972da64675e22b32467e59bbdb0074470da0dc844ebb3000cdabeac89ed179a1: Status 404 returned error can't find the container with id 972da64675e22b32467e59bbdb0074470da0dc844ebb3000cdabeac89ed179a1 Nov 01 22:28:26.424509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.424482 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:28:26.685668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:26.685637 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:28:26.686244 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:26.686207 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe270464_ebdc_46e4_b575_e1884a42d986.slice/crio-864f370cd3d75850c020459dffa88d6311af66d72f6ea47738b4b9e9aef1f8c3 WatchSource:0}: Error finding container 864f370cd3d75850c020459dffa88d6311af66d72f6ea47738b4b9e9aef1f8c3: Status 404 returned error can't find the container with id 864f370cd3d75850c020459dffa88d6311af66d72f6ea47738b4b9e9aef1f8c3 Nov 01 22:28:27.187550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.187493 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" event={"ID":"382b2917-3446-498f-97d9-0a56ae83cd24","Type":"ContainerStarted","Data":"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc"} Nov 01 22:28:27.187975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.187572 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" event={"ID":"382b2917-3446-498f-97d9-0a56ae83cd24","Type":"ContainerStarted","Data":"972da64675e22b32467e59bbdb0074470da0dc844ebb3000cdabeac89ed179a1"} Nov 01 22:28:27.188315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.188295 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:27.189282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.189255 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" event={"ID":"fe270464-ebdc-46e4-b575-e1884a42d986","Type":"ContainerStarted","Data":"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3"} Nov 01 22:28:27.189411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.189307 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" event={"ID":"fe270464-ebdc-46e4-b575-e1884a42d986","Type":"ContainerStarted","Data":"864f370cd3d75850c020459dffa88d6311af66d72f6ea47738b4b9e9aef1f8c3"} Nov 01 22:28:27.189411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.189334 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:27.209316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.209269 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" podStartSLOduration=2.209255619 podStartE2EDuration="2.209255619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:27.205146913 +0000 UTC m=+2654.179049453" watchObservedRunningTime="2026-11-01 22:28:27.209255619 +0000 UTC m=+2654.183158180" Nov 01 22:28:27.225901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.225853 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" podStartSLOduration=2.22583783 podStartE2EDuration="2.22583783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:27.223020313 +0000 UTC m=+2654.196922844" watchObservedRunningTime="2026-11-01 22:28:27.22583783 +0000 UTC m=+2654.199740371" Nov 01 22:28:27.244762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.244728 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 22:28:27.255734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:27.255702 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 22:28:28.162914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:28.162888 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:28:29.122505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.122465 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" podUID="12f52ec8-8c56-4bff-9557-33535307f6f4" containerName="main" containerID="cri-o://3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267" gracePeriod=45 Nov 01 22:28:29.406394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.406369 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:28:29.502439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502409 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502449 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502641 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:29.502751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502683 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502753 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502769 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:29.502819 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502796 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502819 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\" (UniqueName: \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\") pod \"12f52ec8-8c56-4bff-9557-33535307f6f4\" (UID: \"12f52ec8-8c56-4bff-9557-33535307f6f4\") " Nov 01 22:28:29.502951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.502938 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:29.503551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.503524 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:29.503551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.503542 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:29.503551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.503552 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:29.504970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.504944 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:29.505128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.505109 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "kube-api-access-pg6hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:29.505177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.505127 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs" pod "12f52ec8-8c56-4bff-9557-33535307f6f4" (UID: "12f52ec8-8c56-4bff-9557-33535307f6f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:29.604875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.604855 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f52ec8-8c56-4bff-9557-33535307f6f4-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:29.604875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.604877 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f52ec8-8c56-4bff-9557-33535307f6f4-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:29.605020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:29.604887 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6hc\" (UniqueName: \"kubernetes.io/projected/12f52ec8-8c56-4bff-9557-33535307f6f4-kube-api-access-pg6hc\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:30.214435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.214396 4395 generic.go:356] "Generic (PLEG): container finished" podID="12f52ec8-8c56-4bff-9557-33535307f6f4" containerID="3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267" exitCode=0 Nov 01 22:28:30.214915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.214454 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" Nov 01 22:28:30.214915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.214477 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" event={"ID":"12f52ec8-8c56-4bff-9557-33535307f6f4","Type":"ContainerDied","Data":"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267"} Nov 01 22:28:30.214915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.214526 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd" event={"ID":"12f52ec8-8c56-4bff-9557-33535307f6f4","Type":"ContainerDied","Data":"a99cd94a32e8ea5eb22c5bfa0929f3dcccb66339554e09124e488a81532a92d2"} Nov 01 22:28:30.214915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.214544 4395 scope.go:122] "RemoveContainer" containerID="3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267" Nov 01 22:28:30.227058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.227019 4395 scope.go:122] "RemoveContainer" containerID="3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267" Nov 01 22:28:30.227330 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:30.227309 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267\": container with ID starting with 3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267 not found: ID does not exist" containerID="3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267" Nov 01 22:28:30.227382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.227339 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267"} err="failed to get container status \"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267\": rpc error: code = NotFound desc = could not find container \"3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267\": container with ID starting with 3dc6429a52342c798add4a94915396c83695540fad01c8480aafcd5dc4e90267 not found: ID does not exist" Nov 01 22:28:30.240973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.240952 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:28:30.244991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:30.244971 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1783e3/sub-test-model-1783e3-kserve-6f657b876c-fqfvd"] Nov 01 22:28:31.169739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.169709 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f52ec8-8c56-4bff-9557-33535307f6f4" path="/var/lib/kubelet/pods/12f52ec8-8c56-4bff-9557-33535307f6f4/volumes" Nov 01 22:28:31.399626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.399596 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n"] Nov 01 22:28:31.400806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.400782 4395 cpu_manager.go:414] "containerMap: removing container" podUID="12f52ec8-8c56-4bff-9557-33535307f6f4" containerName="main" Nov 01 22:28:31.400902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.400808 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f52ec8-8c56-4bff-9557-33535307f6f4" containerName="main" Nov 01 22:28:31.401003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.400989 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f52ec8-8c56-4bff-9557-33535307f6f4" containerName="main" Nov 01 22:28:31.406122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.406094 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.412920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.412890 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n"] Nov 01 22:28:31.467425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.467393 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:28:31.467588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.467532 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:28:31.539273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.539238 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.539413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.539349 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmksq\" (UniqueName: \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.539509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.539493 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.640423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.640395 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmksq\" (UniqueName: \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.640582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.640487 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.640651 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.640638 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.641246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.641226 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.642981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.642960 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.654787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.654753 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmksq\" (UniqueName: \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\") pod \"maas-api-revoke-sub-6801004e-gvc9n\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:31.733934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:31.733884 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:32.071738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:32.071710 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n"] Nov 01 22:28:32.073098 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:32.073067 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20caf657_54b6_4ba0_a793_8284b6e3ec65.slice/crio-7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150 WatchSource:0}: Error finding container 7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150: Status 404 returned error can't find the container with id 7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150 Nov 01 22:28:32.230792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:32.230755 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" event={"ID":"20caf657-54b6-4ba0-a793-8284b6e3ec65","Type":"ContainerStarted","Data":"7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150"} Nov 01 22:28:32.245862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:32.245822 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" containerID="cri-o://65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e" gracePeriod=25 Nov 01 22:28:32.256104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:32.256061 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" containerID="cri-o://935d7e07f8c0a6711b3e020b1be296c0ba9c65a4ee6cec0f1773a7107c73b7e2" gracePeriod=25 Nov 01 22:28:33.239422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:33.239333 4395 generic.go:356] "Generic (PLEG): container finished" podID="20caf657-54b6-4ba0-a793-8284b6e3ec65" containerID="3280bf24cc006a94c08fb597620b9ebc7d00bdcb3629ab3701410f76b7c1cbfb" exitCode=0 Nov 01 22:28:33.239804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:33.239411 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" event={"ID":"20caf657-54b6-4ba0-a793-8284b6e3ec65","Type":"ContainerDied","Data":"3280bf24cc006a94c08fb597620b9ebc7d00bdcb3629ab3701410f76b7c1cbfb"} Nov 01 22:28:34.424889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.424861 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:34.531679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.531650 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\") pod \"20caf657-54b6-4ba0-a793-8284b6e3ec65\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " Nov 01 22:28:34.531843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.531697 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\") pod \"20caf657-54b6-4ba0-a793-8284b6e3ec65\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " Nov 01 22:28:34.531843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.531780 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\" (UniqueName: \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\") pod \"20caf657-54b6-4ba0-a793-8284b6e3ec65\" (UID: \"20caf657-54b6-4ba0-a793-8284b6e3ec65\") " Nov 01 22:28:34.532151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.532126 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca" pod "20caf657-54b6-4ba0-a793-8284b6e3ec65" (UID: "20caf657-54b6-4ba0-a793-8284b6e3ec65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:34.532551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.532529 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:34.534002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.533970 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca" pod "20caf657-54b6-4ba0-a793-8284b6e3ec65" (UID: "20caf657-54b6-4ba0-a793-8284b6e3ec65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:34.534120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.534069 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq" pod "20caf657-54b6-4ba0-a793-8284b6e3ec65" (UID: "20caf657-54b6-4ba0-a793-8284b6e3ec65"). InnerVolumeSpecName "kube-api-access-vmksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:34.633705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.633668 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20caf657-54b6-4ba0-a793-8284b6e3ec65-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:34.633705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.633701 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmksq\" (UniqueName: \"kubernetes.io/projected/20caf657-54b6-4ba0-a793-8284b6e3ec65-kube-api-access-vmksq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:34.684453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:34.684421 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:35.255150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.255105 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" event={"ID":"20caf657-54b6-4ba0-a793-8284b6e3ec65","Type":"ContainerDied","Data":"7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150"} Nov 01 22:28:35.255317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.255152 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n" Nov 01 22:28:35.255317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.255166 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7650049d14b32ff18ddc8bcee0bb62075627d0d9190955f086755e0d0df50150" Nov 01 22:28:35.304925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.304886 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerName="main" containerID="cri-o://b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4" gracePeriod=45 Nov 01 22:28:35.704705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.704666 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:35.705436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.705411 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:35.706527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.706507 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:28:35.751215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751185 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751319 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751387 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751425 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:35.751571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751537 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751594 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\" (UniqueName: \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751648 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\") pod \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\" (UID: \"dbe82684-41ae-4d94-ad7c-8c646fc5b271\") " Nov 01 22:28:35.751777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751596 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:35.751839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.751619 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:35.752390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.752365 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.752390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.752395 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.752581 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.752410 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.754388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.754360 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:35.754388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.754369 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:35.754523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.754505 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945" pod "dbe82684-41ae-4d94-ad7c-8c646fc5b271" (UID: "dbe82684-41ae-4d94-ad7c-8c646fc5b271"). InnerVolumeSpecName "kube-api-access-9k945". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:35.853736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.853710 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe82684-41ae-4d94-ad7c-8c646fc5b271-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.853736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.853733 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k945\" (UniqueName: \"kubernetes.io/projected/dbe82684-41ae-4d94-ad7c-8c646fc5b271-kube-api-access-9k945\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:35.853861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:35.853743 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe82684-41ae-4d94-ad7c-8c646fc5b271-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.265418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.265392 4395 generic.go:356] "Generic (PLEG): container finished" podID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerID="f5da4ed42ecd616cff0c36af64d07f517d4ccdd9ac728cc443e045b3c2a67860" exitCode=0 Nov 01 22:28:36.265521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.265464 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" event={"ID":"56d34f11-33b0-4247-8fbc-ebce0b2a13d7","Type":"ContainerDied","Data":"f5da4ed42ecd616cff0c36af64d07f517d4ccdd9ac728cc443e045b3c2a67860"} Nov 01 22:28:36.267209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.267186 4395 generic.go:356] "Generic (PLEG): container finished" podID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerID="b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4" exitCode=0 Nov 01 22:28:36.267301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.267220 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" event={"ID":"dbe82684-41ae-4d94-ad7c-8c646fc5b271","Type":"ContainerDied","Data":"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4"} Nov 01 22:28:36.267301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.267228 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" Nov 01 22:28:36.267301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.267248 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" event={"ID":"dbe82684-41ae-4d94-ad7c-8c646fc5b271","Type":"ContainerDied","Data":"69a780100e718430d70d8948809b9a8bdbe6fa89497e1946ccc4c4ce8e75dcba"} Nov 01 22:28:36.267301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.267267 4395 scope.go:122] "RemoveContainer" containerID="b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4" Nov 01 22:28:36.278623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.278598 4395 scope.go:122] "RemoveContainer" containerID="b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4" Nov 01 22:28:36.279578 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:36.279554 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4\": container with ID starting with b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4 not found: ID does not exist" containerID="b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4" Nov 01 22:28:36.279673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.279586 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4"} err="failed to get container status \"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4\": rpc error: code = NotFound desc = could not find container \"b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4\": container with ID starting with b236f4c6c154470d9822c9294a139c44c991cb0f892cc1e9b84a2e4f785da4c4 not found: ID does not exist" Nov 01 22:28:36.292616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.292594 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:28:36.301013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.300991 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79"] Nov 01 22:28:36.421021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.421000 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 22:28:36.463004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.462976 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\" (UniqueName: \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\") pod \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " Nov 01 22:28:36.463191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.463089 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\") pod \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\" (UID: \"56d34f11-33b0-4247-8fbc-ebce0b2a13d7\") " Nov 01 22:28:36.463532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.463504 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume" pod "56d34f11-33b0-4247-8fbc-ebce0b2a13d7" (UID: "56d34f11-33b0-4247-8fbc-ebce0b2a13d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:36.463960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.463930 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.465323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.465302 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2" pod "56d34f11-33b0-4247-8fbc-ebce0b2a13d7" (UID: "56d34f11-33b0-4247-8fbc-ebce0b2a13d7"). InnerVolumeSpecName "kube-api-access-2s9s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:36.565721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.565692 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9s2\" (UniqueName: \"kubernetes.io/projected/56d34f11-33b0-4247-8fbc-ebce0b2a13d7-kube-api-access-2s9s2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:36.629584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.629548 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-e2bf63/sub-test-model-e2bf63-kserve-7cbd7dfb4f-zds79" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": context deadline exceeded" Nov 01 22:28:36.873894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:36.873859 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" containerID="cri-o://f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e" gracePeriod=45 Nov 01 22:28:37.161569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.161544 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:28:37.166654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.166624 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" path="/var/lib/kubelet/pods/dbe82684-41ae-4d94-ad7c-8c646fc5b271/volumes" Nov 01 22:28:37.278205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.278173 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" Nov 01 22:28:37.278424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.278196 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n" event={"ID":"56d34f11-33b0-4247-8fbc-ebce0b2a13d7","Type":"ContainerDied","Data":"1581564ff501de85ece172fa1de64946cf671178feb80c328004b62601b8b294"} Nov 01 22:28:37.278424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.278300 4395 scope.go:122] "RemoveContainer" containerID="f5da4ed42ecd616cff0c36af64d07f517d4ccdd9ac728cc443e045b3c2a67860" Nov 01 22:28:37.280531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.280500 4395 generic.go:356] "Generic (PLEG): container finished" podID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerID="f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e" exitCode=0 Nov 01 22:28:37.280709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.280689 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" event={"ID":"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1","Type":"ContainerDied","Data":"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e"} Nov 01 22:28:37.280787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.280730 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" event={"ID":"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1","Type":"ContainerDied","Data":"743634a450b144f9d1b612931f1ccd33c8ed1de15bfceebdd7d5a5f7e79b7d7b"} Nov 01 22:28:37.280853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.280809 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp" Nov 01 22:28:37.286491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.286314 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.286849 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.286988 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287049 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287116 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287162 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\" (UniqueName: \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287274 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\") pod \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\" (UID: \"a6a36ebb-0113-4752-a28e-6ab0c15f5fd1\") " Nov 01 22:28:37.287809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287382 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:37.287874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.287851 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:37.289252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.288044 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.289252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.288071 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.289252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.288086 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.294494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.294468 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "kube-api-access-4sm5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:37.294639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.294500 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:37.295097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.295068 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm" pod "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" (UID: "a6a36ebb-0113-4752-a28e-6ab0c15f5fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:28:37.296709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.296679 4395 scope.go:122] "RemoveContainer" containerID="f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e" Nov 01 22:28:37.320850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.320831 4395 scope.go:122] "RemoveContainer" containerID="f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e" Nov 01 22:28:37.321280 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:37.321248 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e\": container with ID starting with f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e not found: ID does not exist" containerID="f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e" Nov 01 22:28:37.321384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.321290 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e"} err="failed to get container status \"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e\": rpc error: code = NotFound desc = could not find container \"f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e\": container with ID starting with f6f00a6ced7fe100b67cb91b14767ba2c1dd01c0003aaf2c2882227f8fcfad9e not found: ID does not exist" Nov 01 22:28:37.333565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.333526 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 22:28:37.338627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.338601 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-fb9597-86489d9d58-t5b2n"] Nov 01 22:28:37.392066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.391988 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.392066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.392021 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.392066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.392056 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sm5x\" (UniqueName: \"kubernetes.io/projected/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1-kube-api-access-4sm5x\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:37.675051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.669978 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 22:28:37.686058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:37.684230 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/distinct-2-w2-f033b1-kserve-f88cd889-z4xqp"] Nov 01 22:28:38.201213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:38.201186 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:28:39.170253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:39.170220 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" path="/var/lib/kubelet/pods/56d34f11-33b0-4247-8fbc-ebce0b2a13d7/volumes" Nov 01 22:28:39.170910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:39.170891 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" path="/var/lib/kubelet/pods/a6a36ebb-0113-4752-a28e-6ab0c15f5fd1/volumes" Nov 01 22:28:39.207827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:39.207803 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:28:39.470846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:39.470773 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:40.336521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.336487 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:28:40.337308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337290 4395 cpu_manager.go:414] "containerMap: removing container" podUID="20caf657-54b6-4ba0-a793-8284b6e3ec65" containerName="revoke-keys" Nov 01 22:28:40.337308 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337309 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20caf657-54b6-4ba0-a793-8284b6e3ec65" containerName="revoke-keys" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337337 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337342 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337348 4395 cpu_manager.go:414] "containerMap: removing container" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerName="main" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337353 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerName="main" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337372 4395 cpu_manager.go:414] "containerMap: removing container" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" Nov 01 22:28:40.337443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337377 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" Nov 01 22:28:40.337619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337470 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d34f11-33b0-4247-8fbc-ebce0b2a13d7" containerName="payload-processing" Nov 01 22:28:40.337619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337482 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="20caf657-54b6-4ba0-a793-8284b6e3ec65" containerName="revoke-keys" Nov 01 22:28:40.337619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337492 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a36ebb-0113-4752-a28e-6ab0c15f5fd1" containerName="main" Nov 01 22:28:40.337619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.337498 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe82684-41ae-4d94-ad7c-8c646fc5b271" containerName="main" Nov 01 22:28:40.341277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.341250 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.349150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.349109 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c6bb56-serving-cert\"" Nov 01 22:28:40.355903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.355877 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:28:40.474072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.474008 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnffm\" (UniqueName: \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.474221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.474186 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.474221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.474216 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.474301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.474243 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.577049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.576147 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.577049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.576208 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.577049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.576257 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.577049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.576513 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnffm\" (UniqueName: \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.583168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.577958 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.587098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.584295 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.587098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.584599 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.593249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.591877 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnffm\" (UniqueName: \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\") pod \"maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.667276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.667248 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:40.999782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:40.999741 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:28:41.134061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.134003 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:28:41.135890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.135856 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" podUID="7ff121c3-177d-4a67-8f13-98da0bde9334" containerName="payload-pre-processing" containerID="cri-o://acc9e533505146d5d850cc7400820b80d9b271a678f837d1ab3cacc66954faf2" gracePeriod=30 Nov 01 22:28:41.424363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.423940 4395 generic.go:356] "Generic (PLEG): container finished" podID="7ff121c3-177d-4a67-8f13-98da0bde9334" containerID="acc9e533505146d5d850cc7400820b80d9b271a678f837d1ab3cacc66954faf2" exitCode=0 Nov 01 22:28:41.424363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.424269 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" event={"ID":"7ff121c3-177d-4a67-8f13-98da0bde9334","Type":"ContainerDied","Data":"acc9e533505146d5d850cc7400820b80d9b271a678f837d1ab3cacc66954faf2"} Nov 01 22:28:41.431053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.431000 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" event={"ID":"6b55ddbc-379c-412b-a308-5162592a305d","Type":"ContainerStarted","Data":"7630b8dbbc318d8d250a09ac880fa2591b531d02c123f7e55d1555de01bcfc42"} Nov 01 22:28:41.688403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.688370 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:28:41.786298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.781083 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\") pod \"7ff121c3-177d-4a67-8f13-98da0bde9334\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " Nov 01 22:28:41.786298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.781212 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\" (UniqueName: \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\") pod \"7ff121c3-177d-4a67-8f13-98da0bde9334\" (UID: \"7ff121c3-177d-4a67-8f13-98da0bde9334\") " Nov 01 22:28:41.804517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.804477 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume" pod "7ff121c3-177d-4a67-8f13-98da0bde9334" (UID: "7ff121c3-177d-4a67-8f13-98da0bde9334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:41.833288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.833245 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r" pod "7ff121c3-177d-4a67-8f13-98da0bde9334" (UID: "7ff121c3-177d-4a67-8f13-98da0bde9334"). InnerVolumeSpecName "kube-api-access-b8d4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:41.883284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.883256 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff121c3-177d-4a67-8f13-98da0bde9334-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:41.883284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:41.883282 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8d4r\" (UniqueName: \"kubernetes.io/projected/7ff121c3-177d-4a67-8f13-98da0bde9334-kube-api-access-b8d4r\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:42.472693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.472650 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" event={"ID":"6b55ddbc-379c-412b-a308-5162592a305d","Type":"ContainerStarted","Data":"83b836b3b75eeb673d334346ce4fd308fd5a323b535bf80fcd366885b8b18b05"} Nov 01 22:28:42.475407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.474541 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:42.479621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.479578 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" event={"ID":"7ff121c3-177d-4a67-8f13-98da0bde9334","Type":"ContainerDied","Data":"2d74151a25b726274bc5c010de97ad2586515334996bc70f4beab18b52c69db3"} Nov 01 22:28:42.479823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.479793 4395 scope.go:122] "RemoveContainer" containerID="acc9e533505146d5d850cc7400820b80d9b271a678f837d1ab3cacc66954faf2" Nov 01 22:28:42.480125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.480108 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh" Nov 01 22:28:42.513234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.513183 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" podStartSLOduration=1.993349268 podStartE2EDuration="2.513166738s" totalImagesPullingTime="519.81747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:42.498746678 +0000 UTC m=+2669.472649218" watchObservedRunningTime="2026-11-01 22:28:42.513166738 +0000 UTC m=+2669.487069276" Nov 01 22:28:42.530269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.530211 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:28:42.534229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:42.534208 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a00c6a-66fcc5cdcf-zstfh"] Nov 01 22:28:43.233050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:43.231082 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff121c3-177d-4a67-8f13-98da0bde9334" path="/var/lib/kubelet/pods/7ff121c3-177d-4a67-8f13-98da0bde9334/volumes" Nov 01 22:28:44.471779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.471737 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerName="payload-processing" containerID="cri-o://3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e" gracePeriod=25 Nov 01 22:28:44.618501 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:44.618476 4395 logging.go:55] [core] [Channel #11240 SubChannel #11241]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.224:50052", ServerName: "10.133.0.224:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.224:50052: connect: connection refused" Nov 01 22:28:44.804741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.803494 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:44.893992 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.893965 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\") pod \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " Nov 01 22:28:44.894242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.894229 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\" (UniqueName: \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\") pod \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\" (UID: \"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358\") " Nov 01 22:28:44.894371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.894343 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume" pod "8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" (UID: "8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:44.894701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.894685 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:44.896498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.896477 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw" pod "8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" (UID: "8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358"). InnerVolumeSpecName "kube-api-access-b4msw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:44.995849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:44.995795 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4msw\" (UniqueName: \"kubernetes.io/projected/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358-kube-api-access-b4msw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:45.509614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.509578 4395 generic.go:356] "Generic (PLEG): container finished" podID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerID="3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e" exitCode=0 Nov 01 22:28:45.510221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.509628 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" Nov 01 22:28:45.510221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.509674 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" event={"ID":"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358","Type":"ContainerDied","Data":"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e"} Nov 01 22:28:45.510221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.509739 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" event={"ID":"8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358","Type":"ContainerDied","Data":"4e59e684cfcdc3ef6efce03a257b3e657beab66cc9b8c335f1c5f28295ad90ec"} Nov 01 22:28:45.510221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.509774 4395 scope.go:122] "RemoveContainer" containerID="3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e" Nov 01 22:28:45.520514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.520499 4395 scope.go:122] "RemoveContainer" containerID="3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e" Nov 01 22:28:45.520781 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:45.520764 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e\": container with ID starting with 3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e not found: ID does not exist" containerID="3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e" Nov 01 22:28:45.520830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.520790 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e"} err="failed to get container status \"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e\": rpc error: code = NotFound desc = could not find container \"3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e\": container with ID starting with 3b0cfd077016a380ea7f498f8613c1b2c79eb3fbbd1677fb02a17a05eecc0a2e not found: ID does not exist" Nov 01 22:28:45.530597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.530577 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:45.537068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.537048 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58"] Nov 01 22:28:45.620049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.619185 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-769b6dff6d-zrq58" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.224:50052\" within 1s: context deadline exceeded" Nov 01 22:28:45.704646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.704597 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:45.704646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:45.704631 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:28:47.208225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:47.207619 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" path="/var/lib/kubelet/pods/8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358/volumes" Nov 01 22:28:49.507837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:49.507807 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:28:52.321758 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:52.321623 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:28:52.484340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.483973 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 22:28:52.555981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.555898 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\" (UniqueName: \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\") pod \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " Nov 01 22:28:52.556178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.556064 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\") pod \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\" (UID: \"5261cb1c-37ea-43a6-8edd-55e1c3f624e2\") " Nov 01 22:28:52.557346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.557317 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume" pod "5261cb1c-37ea-43a6-8edd-55e1c3f624e2" (UID: "5261cb1c-37ea-43a6-8edd-55e1c3f624e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:52.559182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.559156 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9" pod "5261cb1c-37ea-43a6-8edd-55e1c3f624e2" (UID: "5261cb1c-37ea-43a6-8edd-55e1c3f624e2"). InnerVolumeSpecName "kube-api-access-655q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:52.563663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.563636 4395 generic.go:356] "Generic (PLEG): container finished" podID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerID="935d7e07f8c0a6711b3e020b1be296c0ba9c65a4ee6cec0f1773a7107c73b7e2" exitCode=0 Nov 01 22:28:52.563760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.563683 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" event={"ID":"6238bac1-202d-49f0-9c48-3bc87b9158ae","Type":"ContainerDied","Data":"935d7e07f8c0a6711b3e020b1be296c0ba9c65a4ee6cec0f1773a7107c73b7e2"} Nov 01 22:28:52.565437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.565414 4395 generic.go:356] "Generic (PLEG): container finished" podID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerID="65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e" exitCode=0 Nov 01 22:28:52.565541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.565460 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" Nov 01 22:28:52.565541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.565498 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" event={"ID":"5261cb1c-37ea-43a6-8edd-55e1c3f624e2","Type":"ContainerDied","Data":"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e"} Nov 01 22:28:52.565648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.565556 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt" event={"ID":"5261cb1c-37ea-43a6-8edd-55e1c3f624e2","Type":"ContainerDied","Data":"4fa672436068c9402f91a7955c3475f9a2240e5e9724244dff905d53df80042a"} Nov 01 22:28:52.565648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.565590 4395 scope.go:122] "RemoveContainer" containerID="65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e" Nov 01 22:28:52.581075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.581056 4395 scope.go:122] "RemoveContainer" containerID="65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e" Nov 01 22:28:52.581396 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:52.581369 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e\": container with ID starting with 65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e not found: ID does not exist" containerID="65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e" Nov 01 22:28:52.581461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.581408 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e"} err="failed to get container status \"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e\": rpc error: code = NotFound desc = could not find container \"65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e\": container with ID starting with 65765747077eb49bb3f86b4577450b1a2013cbec7d9295d6f4ec403f782b528e not found: ID does not exist" Nov 01 22:28:52.585411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.585394 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 22:28:52.599094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.599070 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 22:28:52.607142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.607123 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f033b1-7f46dbb79b-t2ptt"] Nov 01 22:28:52.658090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.658062 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\" (UniqueName: \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\") pod \"6238bac1-202d-49f0-9c48-3bc87b9158ae\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " Nov 01 22:28:52.658247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.658122 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\") pod \"6238bac1-202d-49f0-9c48-3bc87b9158ae\" (UID: \"6238bac1-202d-49f0-9c48-3bc87b9158ae\") " Nov 01 22:28:52.658805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.658778 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655q9\" (UniqueName: \"kubernetes.io/projected/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-kube-api-access-655q9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:52.658805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.658795 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume" pod "6238bac1-202d-49f0-9c48-3bc87b9158ae" (UID: "6238bac1-202d-49f0-9c48-3bc87b9158ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:52.658994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.658809 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261cb1c-37ea-43a6-8edd-55e1c3f624e2-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:52.661099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.661074 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm" pod "6238bac1-202d-49f0-9c48-3bc87b9158ae" (UID: "6238bac1-202d-49f0-9c48-3bc87b9158ae"). InnerVolumeSpecName "kube-api-access-h5lwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:52.759716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.759692 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5lwm\" (UniqueName: \"kubernetes.io/projected/6238bac1-202d-49f0-9c48-3bc87b9158ae-kube-api-access-h5lwm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:52.759813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.759718 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6238bac1-202d-49f0-9c48-3bc87b9158ae-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:52.929398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.920784 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 22:28:52.929398 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:52.921045 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" podUID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" containerName="maas-api" containerID="cri-o://ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08" gracePeriod=30 Nov 01 22:28:53.401296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.398956 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" path="/var/lib/kubelet/pods/5261cb1c-37ea-43a6-8edd-55e1c3f624e2/volumes" Nov 01 22:28:53.500337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.500307 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 22:28:53.557627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.557542 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\") pod \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " Nov 01 22:28:53.557627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.557611 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\") pod \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " Nov 01 22:28:53.557857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.557821 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\" (UniqueName: \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\") pod \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " Nov 01 22:28:53.557980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.557961 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") pod \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\" (UID: \"20b9c9f5-f9b2-403f-90b2-f484b4e9a735\") " Nov 01 22:28:53.559109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.558390 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca" pod "20b9c9f5-f9b2-403f-90b2-f484b4e9a735" (UID: "20b9c9f5-f9b2-403f-90b2-f484b4e9a735"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:28:53.559109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.558737 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:53.563141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.563112 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls" pod "20b9c9f5-f9b2-403f-90b2-f484b4e9a735" (UID: "20b9c9f5-f9b2-403f-90b2-f484b4e9a735"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:53.563339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.563321 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh" pod "20b9c9f5-f9b2-403f-90b2-f484b4e9a735" (UID: "20b9c9f5-f9b2-403f-90b2-f484b4e9a735"). InnerVolumeSpecName "kube-api-access-whrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:53.567652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.567626 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs" pod "20b9c9f5-f9b2-403f-90b2-f484b4e9a735" (UID: "20b9c9f5-f9b2-403f-90b2-f484b4e9a735"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:28:53.582555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.582526 4395 generic.go:356] "Generic (PLEG): container finished" podID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" containerID="ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08" exitCode=0 Nov 01 22:28:53.582748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.582612 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" Nov 01 22:28:53.582748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.582721 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" event={"ID":"20b9c9f5-f9b2-403f-90b2-f484b4e9a735","Type":"ContainerDied","Data":"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08"} Nov 01 22:28:53.582892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.582808 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt" event={"ID":"20b9c9f5-f9b2-403f-90b2-f484b4e9a735","Type":"ContainerDied","Data":"0d27022c20e492529ab3e1fa3ee919becea92b9bd1b5ddbfc19bd8c184d7892b"} Nov 01 22:28:53.582892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.582842 4395 scope.go:122] "RemoveContainer" containerID="ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08" Nov 01 22:28:53.591616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.591585 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" event={"ID":"6238bac1-202d-49f0-9c48-3bc87b9158ae","Type":"ContainerDied","Data":"b1213e4bf2aaafd3bbe934422c2e8eb3dc70adcd654ed211a4ab771d4d8ec996"} Nov 01 22:28:53.591745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.591712 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9" Nov 01 22:28:53.602186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.602167 4395 scope.go:122] "RemoveContainer" containerID="ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08" Nov 01 22:28:53.602500 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:28:53.602468 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08\": container with ID starting with ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08 not found: ID does not exist" containerID="ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08" Nov 01 22:28:53.602611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.602511 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08"} err="failed to get container status \"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08\": rpc error: code = NotFound desc = could not find container \"ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08\": container with ID starting with ab0fd974a98b56ad397689a19a5d06158f63d7091b342adc51d2cac084eeba08 not found: ID does not exist" Nov 01 22:28:53.602611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.602547 4395 scope.go:122] "RemoveContainer" containerID="935d7e07f8c0a6711b3e020b1be296c0ba9c65a4ee6cec0f1773a7107c73b7e2" Nov 01 22:28:53.627492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.627466 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 22:28:53.631421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.631400 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f033b1-84ccb79598-4zjbt"] Nov 01 22:28:53.646716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.646665 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 22:28:53.650236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.650215 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f033b1-54b598887f-rmgb9"] Nov 01 22:28:53.659810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.659790 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:53.659887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.659813 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:53.659887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:53.659824 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrnh\" (UniqueName: \"kubernetes.io/projected/20b9c9f5-f9b2-403f-90b2-f484b4e9a735-kube-api-access-whrnh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:54.450009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.449980 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:28:54.450708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450691 4395 cpu_manager.go:414] "containerMap: removing container" podUID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" containerName="maas-api" Nov 01 22:28:54.450766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450716 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" containerName="maas-api" Nov 01 22:28:54.450766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450735 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" Nov 01 22:28:54.450766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450741 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" Nov 01 22:28:54.450766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450759 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7ff121c3-177d-4a67-8f13-98da0bde9334" containerName="payload-pre-processing" Nov 01 22:28:54.450766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450765 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff121c3-177d-4a67-8f13-98da0bde9334" containerName="payload-pre-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450781 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450786 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450798 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerName="payload-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450803 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerName="payload-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450897 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff121c3-177d-4a67-8f13-98da0bde9334" containerName="payload-pre-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450913 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" containerName="payload-pre-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450924 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6cd6c6-68d8-4ce7-81da-50e5c0b9a358" containerName="payload-processing" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450934 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" containerName="maas-api" Nov 01 22:28:54.450945 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.450941 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5261cb1c-37ea-43a6-8edd-55e1c3f624e2" containerName="payload-processing" Nov 01 22:28:54.456212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.456193 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.462852 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.462826 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db715b\"" Nov 01 22:28:54.463039 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.462904 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db715b-dockercfg-mldbb\"" Nov 01 22:28:54.463417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.463396 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:28:54.490524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.490497 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:28:54.494724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.494700 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.505752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.505730 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:28:54.592054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.581181 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.592054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.581266 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thvw\" (UniqueName: \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.592054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.581392 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wh44\" (UniqueName: \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.592054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.581536 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.708303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.687497 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.708303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.687580 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thvw\" (UniqueName: \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.708303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.687688 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wh44\" (UniqueName: \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.708303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.687818 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.715865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.715837 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wh44\" (UniqueName: \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.763049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.762109 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:54.765282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.763358 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.768594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.768541 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thvw\" (UniqueName: \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\") pod \"payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.775173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.775134 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:54.813969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:54.813923 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:55.073483 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.073452 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:28:55.073646 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:55.073623 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1188269_5865_431f_8592_026f9ea1d72d.slice/crio-28c37e4e3362830878d7985d3ed59a023b1aa0c606f0d85b6e131fd30f465549 WatchSource:0}: Error finding container 28c37e4e3362830878d7985d3ed59a023b1aa0c606f0d85b6e131fd30f465549: Status 404 returned error can't find the container with id 28c37e4e3362830878d7985d3ed59a023b1aa0c606f0d85b6e131fd30f465549 Nov 01 22:28:55.174357 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:28:55.174310 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3218318a_e96a_4031_b5b0_1ad4e17fd2d1.slice/crio-e4aa84c0084a34f01e4acf864e3f57d710115b65fc5d43d1a61b5fbe209d73f3 WatchSource:0}: Error finding container e4aa84c0084a34f01e4acf864e3f57d710115b65fc5d43d1a61b5fbe209d73f3: Status 404 returned error can't find the container with id e4aa84c0084a34f01e4acf864e3f57d710115b65fc5d43d1a61b5fbe209d73f3 Nov 01 22:28:55.187372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.187349 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b9c9f5-f9b2-403f-90b2-f484b4e9a735" path="/var/lib/kubelet/pods/20b9c9f5-f9b2-403f-90b2-f484b4e9a735/volumes" Nov 01 22:28:55.187943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.187924 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6238bac1-202d-49f0-9c48-3bc87b9158ae" path="/var/lib/kubelet/pods/6238bac1-202d-49f0-9c48-3bc87b9158ae/volumes" Nov 01 22:28:55.188368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.188324 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:28:55.765989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.765951 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" event={"ID":"e1188269-5865-431f-8592-026f9ea1d72d","Type":"ContainerStarted","Data":"6e14a311f7a9f0469d53dce70c1cc17dd069fa9e2026614c7fe4af7787b4d803"} Nov 01 22:28:55.766495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.766018 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" event={"ID":"e1188269-5865-431f-8592-026f9ea1d72d","Type":"ContainerStarted","Data":"28c37e4e3362830878d7985d3ed59a023b1aa0c606f0d85b6e131fd30f465549"} Nov 01 22:28:55.766495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.766226 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:28:55.767630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.767603 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" event={"ID":"3218318a-e96a-4031-b5b0-1ad4e17fd2d1","Type":"ContainerStarted","Data":"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28"} Nov 01 22:28:55.767751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.767651 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" event={"ID":"3218318a-e96a-4031-b5b0-1ad4e17fd2d1","Type":"ContainerStarted","Data":"e4aa84c0084a34f01e4acf864e3f57d710115b65fc5d43d1a61b5fbe209d73f3"} Nov 01 22:28:55.767751 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.767672 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:28:55.789999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.789949 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" podStartSLOduration=1.7899364979999999 podStartE2EDuration="1.789936498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:55.784496699 +0000 UTC m=+2682.758399239" watchObservedRunningTime="2026-11-01 22:28:55.789936498 +0000 UTC m=+2682.763839037" Nov 01 22:28:55.807463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:55.807413 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" podStartSLOduration=1.807403078 podStartE2EDuration="1.807403078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:55.802244266 +0000 UTC m=+2682.776146822" watchObservedRunningTime="2026-11-01 22:28:55.807403078 +0000 UTC m=+2682.781305646" Nov 01 22:28:59.723491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:59.723457 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:28:59.736365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:28:59.736330 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:29:04.724742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:04.724693 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" podUID="fe270464-ebdc-46e4-b575-e1884a42d986" containerName="payload-processing" containerID="cri-o://9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3" gracePeriod=25 Nov 01 22:29:04.737331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:04.737283 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" podUID="382b2917-3446-498f-97d9-0a56ae83cd24" containerName="payload-pre-processing" containerID="cri-o://ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc" gracePeriod=25 Nov 01 22:29:05.074715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.074684 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:29:05.144424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.144398 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:29:05.217119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.217078 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\") pod \"fe270464-ebdc-46e4-b575-e1884a42d986\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " Nov 01 22:29:05.217304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.217277 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\" (UniqueName: \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\") pod \"fe270464-ebdc-46e4-b575-e1884a42d986\" (UID: \"fe270464-ebdc-46e4-b575-e1884a42d986\") " Nov 01 22:29:05.217490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.217463 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume" pod "fe270464-ebdc-46e4-b575-e1884a42d986" (UID: "fe270464-ebdc-46e4-b575-e1884a42d986"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:05.217956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.217934 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe270464-ebdc-46e4-b575-e1884a42d986-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:05.219978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.219948 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw" pod "fe270464-ebdc-46e4-b575-e1884a42d986" (UID: "fe270464-ebdc-46e4-b575-e1884a42d986"). InnerVolumeSpecName "kube-api-access-gwrmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:05.319008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.318939 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\" (UniqueName: \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\") pod \"382b2917-3446-498f-97d9-0a56ae83cd24\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " Nov 01 22:29:05.319218 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.319200 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\") pod \"382b2917-3446-498f-97d9-0a56ae83cd24\" (UID: \"382b2917-3446-498f-97d9-0a56ae83cd24\") " Nov 01 22:29:05.319546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.319508 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume" pod "382b2917-3446-498f-97d9-0a56ae83cd24" (UID: "382b2917-3446-498f-97d9-0a56ae83cd24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:05.319746 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.319726 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwrmw\" (UniqueName: \"kubernetes.io/projected/fe270464-ebdc-46e4-b575-e1884a42d986-kube-api-access-gwrmw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:05.319792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.319752 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382b2917-3446-498f-97d9-0a56ae83cd24-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:05.321181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.321157 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff" pod "382b2917-3446-498f-97d9-0a56ae83cd24" (UID: "382b2917-3446-498f-97d9-0a56ae83cd24"). InnerVolumeSpecName "kube-api-access-58pff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:05.420899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.420873 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pff\" (UniqueName: \"kubernetes.io/projected/382b2917-3446-498f-97d9-0a56ae83cd24-kube-api-access-58pff\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:05.841388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.841350 4395 generic.go:356] "Generic (PLEG): container finished" podID="382b2917-3446-498f-97d9-0a56ae83cd24" containerID="ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc" exitCode=0 Nov 01 22:29:05.841914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.841401 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" Nov 01 22:29:05.841914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.841432 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" event={"ID":"382b2917-3446-498f-97d9-0a56ae83cd24","Type":"ContainerDied","Data":"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc"} Nov 01 22:29:05.841914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.841493 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn" event={"ID":"382b2917-3446-498f-97d9-0a56ae83cd24","Type":"ContainerDied","Data":"972da64675e22b32467e59bbdb0074470da0dc844ebb3000cdabeac89ed179a1"} Nov 01 22:29:05.841914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.841521 4395 scope.go:122] "RemoveContainer" containerID="ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc" Nov 01 22:29:05.843665 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.843640 4395 generic.go:356] "Generic (PLEG): container finished" podID="fe270464-ebdc-46e4-b575-e1884a42d986" containerID="9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3" exitCode=0 Nov 01 22:29:05.843775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.843717 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" event={"ID":"fe270464-ebdc-46e4-b575-e1884a42d986","Type":"ContainerDied","Data":"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3"} Nov 01 22:29:05.843775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.843757 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" event={"ID":"fe270464-ebdc-46e4-b575-e1884a42d986","Type":"ContainerDied","Data":"864f370cd3d75850c020459dffa88d6311af66d72f6ea47738b4b9e9aef1f8c3"} Nov 01 22:29:05.843775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.843767 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng" Nov 01 22:29:05.857278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.856296 4395 scope.go:122] "RemoveContainer" containerID="ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc" Nov 01 22:29:05.857278 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:05.856927 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc\": container with ID starting with ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc not found: ID does not exist" containerID="ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc" Nov 01 22:29:05.857278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.856961 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc"} err="failed to get container status \"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc\": rpc error: code = NotFound desc = could not find container \"ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc\": container with ID starting with ad8f7986cd2c7e3782398c2181c67ea83677f72f4d0d43eba9dcfb2b03edb2cc not found: ID does not exist" Nov 01 22:29:05.857278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.856981 4395 scope.go:122] "RemoveContainer" containerID="9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3" Nov 01 22:29:05.867766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.867748 4395 scope.go:122] "RemoveContainer" containerID="9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3" Nov 01 22:29:05.868062 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:05.868013 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3\": container with ID starting with 9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3 not found: ID does not exist" containerID="9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3" Nov 01 22:29:05.868126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.868059 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3"} err="failed to get container status \"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3\": rpc error: code = NotFound desc = could not find container \"9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3\": container with ID starting with 9377be3bd8f4700c2ad2c0fe701eb6414196bd654a1741cba56a98a2b0cea5e3 not found: ID does not exist" Nov 01 22:29:05.876115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.876090 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:29:05.881683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.881661 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-bdc7d9dd8-8b2ng"] Nov 01 22:29:05.892102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.892081 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:29:05.896134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:05.896113 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-64d4f96499-xqzfn"] Nov 01 22:29:06.776688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:06.776660 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:29:06.776988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:06.776952 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:29:07.166765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:07.166680 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382b2917-3446-498f-97d9-0a56ae83cd24" path="/var/lib/kubelet/pods/382b2917-3446-498f-97d9-0a56ae83cd24/volumes" Nov 01 22:29:07.167196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:07.167178 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe270464-ebdc-46e4-b575-e1884a42d986" path="/var/lib/kubelet/pods/fe270464-ebdc-46e4-b575-e1884a42d986/volumes" Nov 01 22:29:13.985081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.985049 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:29:13.985977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.985952 4395 cpu_manager.go:414] "containerMap: removing container" podUID="382b2917-3446-498f-97d9-0a56ae83cd24" containerName="payload-pre-processing" Nov 01 22:29:13.986150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.985978 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382b2917-3446-498f-97d9-0a56ae83cd24" containerName="payload-pre-processing" Nov 01 22:29:13.986150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.986050 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fe270464-ebdc-46e4-b575-e1884a42d986" containerName="payload-processing" Nov 01 22:29:13.986150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.986061 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe270464-ebdc-46e4-b575-e1884a42d986" containerName="payload-processing" Nov 01 22:29:13.986322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.986244 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="382b2917-3446-498f-97d9-0a56ae83cd24" containerName="payload-pre-processing" Nov 01 22:29:13.986322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.986269 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe270464-ebdc-46e4-b575-e1884a42d986" containerName="payload-processing" Nov 01 22:29:13.990309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.990288 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:13.997881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:13.997852 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6bb56-dockercfg-r4w4h\"" Nov 01 22:29:14.003601 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.003573 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:29:14.018058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.016903 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vld57\" (UniqueName: \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.018058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.016984 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.029957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.029928 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:29:14.039289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.039267 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.040832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.040559 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:29:14.049939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.049918 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6bb56\"" Nov 01 22:29:14.118480 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.118443 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.118643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.118606 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vld57\" (UniqueName: \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.118733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.118693 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.118794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.118767 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hn8n\" (UniqueName: \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.119515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.119489 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.137256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.137224 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vld57\" (UniqueName: \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\") pod \"payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.247101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.229146 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hn8n\" (UniqueName: \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.251454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.251412 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.273053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.252661 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.295052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.276777 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hn8n\" (UniqueName: \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\") pod \"payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.309050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.307549 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.365476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.365444 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:14.818936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.813696 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:29:14.933202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.933166 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" event={"ID":"33a1ac87-4847-4614-81ff-df3f3e8f258e","Type":"ContainerStarted","Data":"519a25bf03ff225662e6d6eac8adde205b3de8207b183d50e79e78c3a4221796"} Nov 01 22:29:14.933358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.933231 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" event={"ID":"33a1ac87-4847-4614-81ff-df3f3e8f258e","Type":"ContainerStarted","Data":"3288e4e7948c74ae9349330802e24ed6c6560cbf42fbf95779950a5b0bca58fb"} Nov 01 22:29:14.933358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.933279 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:14.953880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:14.953833 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" podStartSLOduration=1.953817211 podStartE2EDuration="1.953817211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:14.951587 +0000 UTC m=+2701.925489538" watchObservedRunningTime="2026-11-01 22:29:14.953817211 +0000 UTC m=+2701.927719756" Nov 01 22:29:15.038517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:15.038491 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:29:15.039259 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:29:15.039232 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4613ff0c_b459_47ec_94b6_5c8ef1c3b5fc.slice/crio-5c406a6beb82ad17e573a748deb4118c2a4943a4e95b85e7a7fe354d4b01e36d WatchSource:0}: Error finding container 5c406a6beb82ad17e573a748deb4118c2a4943a4e95b85e7a7fe354d4b01e36d: Status 404 returned error can't find the container with id 5c406a6beb82ad17e573a748deb4118c2a4943a4e95b85e7a7fe354d4b01e36d Nov 01 22:29:15.941479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:15.941443 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" event={"ID":"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc","Type":"ContainerStarted","Data":"dc0b65bafd2dade7ed7ff69355daaa8c6b0862254f390aacbf159822360d50fe"} Nov 01 22:29:15.941663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:15.941507 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" event={"ID":"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc","Type":"ContainerStarted","Data":"5c406a6beb82ad17e573a748deb4118c2a4943a4e95b85e7a7fe354d4b01e36d"} Nov 01 22:29:15.941663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:15.941535 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:15.963226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:15.963173 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" podStartSLOduration=2.963161194 podStartE2EDuration="2.963161194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:15.959067445 +0000 UTC m=+2702.932969985" watchObservedRunningTime="2026-11-01 22:29:15.963161194 +0000 UTC m=+2702.937063784" Nov 01 22:29:18.663930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:18.663891 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 22:29:18.664507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:18.664194 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" containerID="cri-o://15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228" gracePeriod=30 Nov 01 22:29:18.990666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:18.990573 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:29:18.990971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:18.990932 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:29:19.543422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:19.543387 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 22:29:19.544018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:19.543986 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" containerID="cri-o://20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb" gracePeriod=30 Nov 01 22:29:19.945563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:19.945476 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:29:19.945563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:19.945536 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:29:22.307175 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:22.307123 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:29:23.889040 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.889004 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 22:29:23.923982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.923945 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924080 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924205 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924412 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924508 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924635 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924669 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.924745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924727 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924801 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924826 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.924997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924905 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\") pod \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\" (UID: \"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f\") " Nov 01 22:29:23.924997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.924927 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.925151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925092 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.925376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925356 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:23.925504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925485 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:23.925571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925512 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:23.925571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925526 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:23.925571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925539 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:23.925571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.925553 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:23.927560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.927533 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:29:23.927696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.927565 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:23.928019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.927992 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "kube-api-access-j75tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:23.928138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:23.928111 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token" pod "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" (UID: "7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:24.050609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.042382 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.050609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.042430 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75tx\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-kube-api-access-j75tx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.050609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.042445 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.050609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.042460 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.123831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.123791 4395 generic.go:356] "Generic (PLEG): container finished" podID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerID="15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228" exitCode=0 Nov 01 22:29:24.124010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.123839 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" Nov 01 22:29:24.124010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.123867 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerDied","Data":"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228"} Nov 01 22:29:24.124010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.123934 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr" event={"ID":"7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f","Type":"ContainerDied","Data":"6eb3415f5b71d6a847f53e2a2fa6007018963296e6c9a7fff0a464e1e98662d9"} Nov 01 22:29:24.124010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.123958 4395 scope.go:122] "RemoveContainer" containerID="15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228" Nov 01 22:29:24.135515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.135496 4395 scope.go:122] "RemoveContainer" containerID="b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.147173 4395 scope.go:122] "RemoveContainer" containerID="15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:24.147456 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228\": container with ID starting with 15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228 not found: ID does not exist" containerID="15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.147480 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228"} err="failed to get container status \"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228\": rpc error: code = NotFound desc = could not find container \"15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228\": container with ID starting with 15272c01d775fd061d425a5a2d7ce8056e39ece02445b0e7a4bb0cfc72316228 not found: ID does not exist" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.147496 4395 scope.go:122] "RemoveContainer" containerID="b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:24.147734 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012\": container with ID starting with b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012 not found: ID does not exist" containerID="b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.147764 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012"} err="failed to get container status \"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012\": rpc error: code = NotFound desc = could not find container \"b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012\": container with ID starting with b5592590d7d589cf60d861487693a35cd03fa46ecc1c2db13a8d29b3931ef012 not found: ID does not exist" Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.150514 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 22:29:24.165738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.154266 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-9p2wr"] Nov 01 22:29:24.198204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.198127 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:29:24.198323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.198199 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:29:24.890683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.890655 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 22:29:24.956152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956117 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956235 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956271 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956316 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956675 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956583 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956595 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956607 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956750 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.956893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.956894 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.957334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.957014 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.957583 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.957561 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\") pod \"f28479be-0e23-471b-bb40-61976623ba89\" (UID: \"f28479be-0e23-471b-bb40-61976623ba89\") " Nov 01 22:29:24.957937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.957261 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.957937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.957455 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.958385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.958360 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.958471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.958392 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.958471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.958408 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28479be-0e23-471b-bb40-61976623ba89-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.958471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.958423 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.958471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.958437 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:24.959190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.959160 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:24.959537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.959506 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:29:24.960560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.960534 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:24.960839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:24.960816 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7" pod "f28479be-0e23-471b-bb40-61976623ba89" (UID: "f28479be-0e23-471b-bb40-61976623ba89"). InnerVolumeSpecName "kube-api-access-v5kh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:25.060442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.060409 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.060442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.060436 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28479be-0e23-471b-bb40-61976623ba89-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.060442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.060446 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28479be-0e23-471b-bb40-61976623ba89-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.060675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.060455 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5kh7\" (UniqueName: \"kubernetes.io/projected/f28479be-0e23-471b-bb40-61976623ba89-kube-api-access-v5kh7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:25.131978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.131889 4395 generic.go:356] "Generic (PLEG): container finished" podID="f28479be-0e23-471b-bb40-61976623ba89" containerID="20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb" exitCode=0 Nov 01 22:29:25.131978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.131933 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" Nov 01 22:29:25.132250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.131970 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" event={"ID":"f28479be-0e23-471b-bb40-61976623ba89","Type":"ContainerDied","Data":"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb"} Nov 01 22:29:25.132250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.132073 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl" event={"ID":"f28479be-0e23-471b-bb40-61976623ba89","Type":"ContainerDied","Data":"7b11832f145fbcd4a3a83fae97939a7a4f2e7f9a468945a2f554bba25f82f910"} Nov 01 22:29:25.132250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.132096 4395 scope.go:122] "RemoveContainer" containerID="20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb" Nov 01 22:29:25.133941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.133918 4395 generic.go:356] "Generic (PLEG): container finished" podID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerID="3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5" exitCode=0 Nov 01 22:29:25.134133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.134000 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerDied","Data":"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5"} Nov 01 22:29:25.134488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.134466 4395 scope.go:122] "RemoveContainer" containerID="3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5" Nov 01 22:29:25.136507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.136471 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.136598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.136564 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.136640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.136613 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:25.145637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.145615 4395 scope.go:122] "RemoveContainer" containerID="20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb" Nov 01 22:29:25.145923 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:25.145902 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb\": container with ID starting with 20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb not found: ID does not exist" containerID="20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb" Nov 01 22:29:25.145985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.145931 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb"} err="failed to get container status \"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb\": rpc error: code = NotFound desc = could not find container \"20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb\": container with ID starting with 20c2ab976781af3d5c205743a2098cbd87c3947c263a4263c0f5fd6105282cbb not found: ID does not exist" Nov 01 22:29:25.178189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.177691 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" path="/var/lib/kubelet/pods/7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f/volumes" Nov 01 22:29:25.180729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.180702 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 22:29:25.195114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.190860 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-rfvbl"] Nov 01 22:29:25.945258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:25.945225 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:29:26.144496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:26.144464 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerStarted","Data":"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349"} Nov 01 22:29:26.598056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:26.597427 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:29:26.598056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:26.597558 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:26.598056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:26.597602 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:26.964104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:26.960205 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:29:27.235279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:27.235095 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28479be-0e23-471b-bb40-61976623ba89" path="/var/lib/kubelet/pods/f28479be-0e23-471b-bb40-61976623ba89/volumes" Nov 01 22:29:27.595054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:27.593382 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:27.595054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:27.593463 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:28.598619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:28.598569 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:28.599138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:28.598633 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:29.604457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:29.604407 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:29.604984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:29.604495 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:30.594057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:30.590354 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:30.594057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:30.590424 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:31.585964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:31.585922 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 22:29:31.586403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:31.585989 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 22:29:32.171742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.171709 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:29:32.172441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172413 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.172441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172434 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.172666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172460 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.172666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172467 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.172666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172488 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" Nov 01 22:29:32.172666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172496 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" Nov 01 22:29:32.172878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172694 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28479be-0e23-471b-bb40-61976623ba89" containerName="istio-proxy" Nov 01 22:29:32.172878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172722 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.172878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.172734 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4399b1-1c58-4e95-bfc3-3cf1dfdc024f" containerName="istio-proxy" Nov 01 22:29:32.179002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.178976 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.197123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.197092 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-60807f-openshift-default-dockercfg-97s8q\"" Nov 01 22:29:32.215530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.215498 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:29:32.263395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263360 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263406 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263487 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263582 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263630 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263745 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263725 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263767 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmctr\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.263869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.263855 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.373595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.373408 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.374911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.374887 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.375479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.375457 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmctr\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.387055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.376046 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.388937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.388910 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.404948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.375380 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.405193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.374770 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.405300 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.403938 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.405472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.405453 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.405626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.405610 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.421088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.405850 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.424351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.424218 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.424775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.424746 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.424869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.422977 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.437866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.437841 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmctr\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.449832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.449799 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.459829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.459791 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.481267 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.481123 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\") pod \"e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.520056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.519397 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:32.592060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.591992 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:29:32.592536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.592377 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:29:32.593240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.593222 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:29:32.849283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.849252 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:29:32.851243 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:29:32.851210 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefce3b9_edf9_402c_a781_2ad03e534677.slice/crio-f7caa3f9344a88b83f6fae0cc973b7ded6176a1d037cccc107f8f1bdc9a13f0b WatchSource:0}: Error finding container f7caa3f9344a88b83f6fae0cc973b7ded6176a1d037cccc107f8f1bdc9a13f0b: Status 404 returned error can't find the container with id f7caa3f9344a88b83f6fae0cc973b7ded6176a1d037cccc107f8f1bdc9a13f0b Nov 01 22:29:32.854188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.854152 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:32.854276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.854224 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:32.854276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:32.854258 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:29:33.326888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:33.326852 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerStarted","Data":"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372"} Nov 01 22:29:33.327158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:33.326912 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerStarted","Data":"f7caa3f9344a88b83f6fae0cc973b7ded6176a1d037cccc107f8f1bdc9a13f0b"} Nov 01 22:29:33.351044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:33.350978 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podStartSLOduration=1.35096345 podStartE2EDuration="1.35096345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:33.34896226 +0000 UTC m=+2720.322864800" watchObservedRunningTime="2026-11-01 22:29:33.35096345 +0000 UTC m=+2720.324866029" Nov 01 22:29:33.520200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:33.520165 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:33.526572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:33.526547 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:34.336069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:34.335135 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:34.336069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:34.335773 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:29:52.466055 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:52.456522 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:29:56.919207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.919114 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:29:56.919704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.919535 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" podUID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" containerName="maas-api" containerID="cri-o://26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9" gracePeriod=30 Nov 01 22:29:56.982755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.982723 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:29:56.982968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.982934 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" podUID="e1188269-5865-431f-8592-026f9ea1d72d" containerName="payload-processing" containerID="cri-o://6e14a311f7a9f0469d53dce70c1cc17dd069fa9e2026614c7fe4af7787b4d803" gracePeriod=30 Nov 01 22:29:56.997747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.997714 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:29:56.998005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:56.997979 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" podUID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" containerName="payload-pre-processing" containerID="cri-o://d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28" gracePeriod=30 Nov 01 22:29:57.536862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.536833 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:29:57.627269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.627190 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\") pod \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " Nov 01 22:29:57.627458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.627347 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\") pod \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " Nov 01 22:29:57.627458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.627381 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\" (UniqueName: \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\") pod \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " Nov 01 22:29:57.627584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.627490 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\") pod \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\" (UID: \"2eb969be-cd66-4fc3-b0c9-4f71025b06ff\") " Nov 01 22:29:57.628479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.628446 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca" pod "2eb969be-cd66-4fc3-b0c9-4f71025b06ff" (UID: "2eb969be-cd66-4fc3-b0c9-4f71025b06ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:57.631430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.631401 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s" pod "2eb969be-cd66-4fc3-b0c9-4f71025b06ff" (UID: "2eb969be-cd66-4fc3-b0c9-4f71025b06ff"). InnerVolumeSpecName "kube-api-access-fq26s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:57.631568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.631421 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs" pod "2eb969be-cd66-4fc3-b0c9-4f71025b06ff" (UID: "2eb969be-cd66-4fc3-b0c9-4f71025b06ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:57.631568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.631435 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls" pod "2eb969be-cd66-4fc3-b0c9-4f71025b06ff" (UID: "2eb969be-cd66-4fc3-b0c9-4f71025b06ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:29:57.645247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.645104 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:29:57.647140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.647104 4395 generic.go:356] "Generic (PLEG): container finished" podID="e1188269-5865-431f-8592-026f9ea1d72d" containerID="6e14a311f7a9f0469d53dce70c1cc17dd069fa9e2026614c7fe4af7787b4d803" exitCode=0 Nov 01 22:29:57.647265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.647170 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" event={"ID":"e1188269-5865-431f-8592-026f9ea1d72d","Type":"ContainerDied","Data":"6e14a311f7a9f0469d53dce70c1cc17dd069fa9e2026614c7fe4af7787b4d803"} Nov 01 22:29:57.648677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.648655 4395 generic.go:356] "Generic (PLEG): container finished" podID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" containerID="26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9" exitCode=0 Nov 01 22:29:57.648799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.648698 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" Nov 01 22:29:57.648799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.648731 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" event={"ID":"2eb969be-cd66-4fc3-b0c9-4f71025b06ff","Type":"ContainerDied","Data":"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9"} Nov 01 22:29:57.648799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.648779 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw" event={"ID":"2eb969be-cd66-4fc3-b0c9-4f71025b06ff","Type":"ContainerDied","Data":"f0729ba6e63cefb6ae70549aac30c49713d7502532158a81ff09e6dac60793e4"} Nov 01 22:29:57.648954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.648808 4395 scope.go:122] "RemoveContainer" containerID="26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9" Nov 01 22:29:57.649270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.649247 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:29:57.650468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.650343 4395 generic.go:356] "Generic (PLEG): container finished" podID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" containerID="d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28" exitCode=0 Nov 01 22:29:57.650468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.650376 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" Nov 01 22:29:57.650468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.650391 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" event={"ID":"3218318a-e96a-4031-b5b0-1ad4e17fd2d1","Type":"ContainerDied","Data":"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28"} Nov 01 22:29:57.650468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.650434 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8" event={"ID":"3218318a-e96a-4031-b5b0-1ad4e17fd2d1","Type":"ContainerDied","Data":"e4aa84c0084a34f01e4acf864e3f57d710115b65fc5d43d1a61b5fbe209d73f3"} Nov 01 22:29:57.661076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.661056 4395 scope.go:122] "RemoveContainer" containerID="26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9" Nov 01 22:29:57.661352 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:57.661332 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9\": container with ID starting with 26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9 not found: ID does not exist" containerID="26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9" Nov 01 22:29:57.661430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.661365 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9"} err="failed to get container status \"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9\": rpc error: code = NotFound desc = could not find container \"26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9\": container with ID starting with 26f4366140b25d7f5d1c80eff51295519173e278e8691dda8c16d2b0c68a70d9 not found: ID does not exist" Nov 01 22:29:57.661430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.661390 4395 scope.go:122] "RemoveContainer" containerID="d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28" Nov 01 22:29:57.742123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.741394 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\" (UniqueName: \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\") pod \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " Nov 01 22:29:57.742123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.741480 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\") pod \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\" (UID: \"3218318a-e96a-4031-b5b0-1ad4e17fd2d1\") " Nov 01 22:29:57.742123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.741667 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\") pod \"e1188269-5865-431f-8592-026f9ea1d72d\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " Nov 01 22:29:57.742123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.741760 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\" (UniqueName: \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\") pod \"e1188269-5865-431f-8592-026f9ea1d72d\" (UID: \"e1188269-5865-431f-8592-026f9ea1d72d\") " Nov 01 22:29:57.742521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.742420 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.742521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.742444 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq26s\" (UniqueName: \"kubernetes.io/projected/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-kube-api-access-fq26s\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.742521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.742459 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.742521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.742472 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eb969be-cd66-4fc3-b0c9-4f71025b06ff-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.746770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.746711 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume" pod "3218318a-e96a-4031-b5b0-1ad4e17fd2d1" (UID: "3218318a-e96a-4031-b5b0-1ad4e17fd2d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:57.748259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.748156 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume" pod "e1188269-5865-431f-8592-026f9ea1d72d" (UID: "e1188269-5865-431f-8592-026f9ea1d72d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:29:57.748259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.748202 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:29:57.756794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.756698 4395 scope.go:122] "RemoveContainer" containerID="d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28" Nov 01 22:29:57.757333 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:29:57.757309 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28\": container with ID starting with d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28 not found: ID does not exist" containerID="d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28" Nov 01 22:29:57.757530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.757507 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28"} err="failed to get container status \"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28\": rpc error: code = NotFound desc = could not find container \"d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28\": container with ID starting with d76243189ae2fde2209e22d64248c6d420b8421f45066215f55c366ace397b28 not found: ID does not exist" Nov 01 22:29:57.760345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.760321 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db715b-78878d8d74-xzznw"] Nov 01 22:29:57.771473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.771441 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44" pod "3218318a-e96a-4031-b5b0-1ad4e17fd2d1" (UID: "3218318a-e96a-4031-b5b0-1ad4e17fd2d1"). InnerVolumeSpecName "kube-api-access-9wh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:57.771886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.771867 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw" pod "e1188269-5865-431f-8592-026f9ea1d72d" (UID: "e1188269-5865-431f-8592-026f9ea1d72d"). InnerVolumeSpecName "kube-api-access-9thvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:57.843981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.843936 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wh44\" (UniqueName: \"kubernetes.io/projected/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-kube-api-access-9wh44\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.843981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.843975 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3218318a-e96a-4031-b5b0-1ad4e17fd2d1-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.843981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.843988 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1188269-5865-431f-8592-026f9ea1d72d-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.844246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.844001 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thvw\" (UniqueName: \"kubernetes.io/projected/e1188269-5865-431f-8592-026f9ea1d72d-kube-api-access-9thvw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:57.979179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.979148 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:29:57.981955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:57.981901 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-794ccf4979-6bdp8"] Nov 01 22:29:58.658363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:58.658334 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" Nov 01 22:29:58.658552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:58.658365 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg" event={"ID":"e1188269-5865-431f-8592-026f9ea1d72d","Type":"ContainerDied","Data":"28c37e4e3362830878d7985d3ed59a023b1aa0c606f0d85b6e131fd30f465549"} Nov 01 22:29:58.658552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:58.658431 4395 scope.go:122] "RemoveContainer" containerID="6e14a311f7a9f0469d53dce70c1cc17dd069fa9e2026614c7fe4af7787b4d803" Nov 01 22:29:58.685314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:58.685286 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:29:58.692420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:58.692392 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db715b-7c7d74974-t8wsg"] Nov 01 22:29:59.167964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:59.167933 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" path="/var/lib/kubelet/pods/2eb969be-cd66-4fc3-b0c9-4f71025b06ff/volumes" Nov 01 22:29:59.168406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:59.168389 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" path="/var/lib/kubelet/pods/3218318a-e96a-4031-b5b0-1ad4e17fd2d1/volumes" Nov 01 22:29:59.168761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:59.168748 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1188269-5865-431f-8592-026f9ea1d72d" path="/var/lib/kubelet/pods/e1188269-5865-431f-8592-026f9ea1d72d/volumes" Nov 01 22:29:59.528694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:59.528659 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:29:59.528956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:29:59.528924 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" containerID="cri-o://10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d" gracePeriod=30 Nov 01 22:30:00.143423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.143387 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:00.144163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144143 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" containerName="maas-api" Nov 01 22:30:00.144226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144166 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" containerName="maas-api" Nov 01 22:30:00.144226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144186 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" containerName="payload-pre-processing" Nov 01 22:30:00.144226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144195 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" containerName="payload-pre-processing" Nov 01 22:30:00.144322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144263 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e1188269-5865-431f-8592-026f9ea1d72d" containerName="payload-processing" Nov 01 22:30:00.144322 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144271 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1188269-5865-431f-8592-026f9ea1d72d" containerName="payload-processing" Nov 01 22:30:00.144444 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144431 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1188269-5865-431f-8592-026f9ea1d72d" containerName="payload-processing" Nov 01 22:30:00.144488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144452 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb969be-cd66-4fc3-b0c9-4f71025b06ff" containerName="maas-api" Nov 01 22:30:00.144488 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.144464 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3218318a-e96a-4031-b5b0-1ad4e17fd2d1" containerName="payload-pre-processing" Nov 01 22:30:00.149810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.149783 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.165349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.165314 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:30:00.169421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.169343 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:00.177661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.176404 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:00.182323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.181894 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.182323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.182182 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gxg\" (UniqueName: \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.182660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.182639 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:00.182813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.182777 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.187721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.187701 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:00.198253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.194529 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.212207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.210743 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.212207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.211131 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:00.231389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.218886 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:00.231389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.226841 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:00.238003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.236592 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:00.250259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.250227 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:00.250548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.250530 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.261870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.258270 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:00.261870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.258313 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:00.261870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.258373 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.284188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.284142 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpv9\" (UniqueName: \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.284363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.284195 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gxg\" (UniqueName: \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.284543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.284523 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtlv\" (UniqueName: \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.284785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.284763 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.284939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.284922 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.285084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.285073 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.285284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.285267 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnsb\" (UniqueName: \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.285538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.285519 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.285729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.285711 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkk7\" (UniqueName: \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.285968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.285950 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.286125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.286106 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.286275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.286259 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.286880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.286858 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.308917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.308894 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gxg\" (UniqueName: \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.387892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.387846 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.387892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.387891 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.388205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.387922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.388205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.387998 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnsb\" (UniqueName: \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.388205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388110 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.388205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388144 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkk7\" (UniqueName: \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.388436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388244 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.388436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388291 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.388436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388335 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpv9\" (UniqueName: \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.388436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388424 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtlv\" (UniqueName: \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.388815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388787 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.388815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388790 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.389060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.388836 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.389179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.389158 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.389416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.389396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.401935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.401860 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.401935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.401863 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtlv\" (UniqueName: \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\") pod \"maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.402146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.401957 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnsb\" (UniqueName: \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\") pod \"maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.402146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.401989 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpv9\" (UniqueName: \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\") pod \"maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.402697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.402679 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkk7\" (UniqueName: \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\") pod \"maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.480103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.480070 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:00.523133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.523093 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:00.539924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.538082 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:00.557394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.555883 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:00.572338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.571723 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:00.583097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.583070 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:00.809409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:00.809157 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:01.008351 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:01.008321 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83889a6b_6f8b_4fb2_bd12_d4ac41ed6931.slice/crio-649f18a15ca6053c9ea75525deea2399092420ab084bd9a8b2a301caeea283d4 WatchSource:0}: Error finding container 649f18a15ca6053c9ea75525deea2399092420ab084bd9a8b2a301caeea283d4: Status 404 returned error can't find the container with id 649f18a15ca6053c9ea75525deea2399092420ab084bd9a8b2a301caeea283d4 Nov 01 22:30:01.008863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.008834 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:01.021536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.021509 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:01.024056 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:01.023995 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0cc2bb_16b1_43e9_b5d8_c18ca2dfdcae.slice/crio-6c436ba0c6855e2fe23e53f6f5609b89288d39036f364dd23803e2d99b9124e3 WatchSource:0}: Error finding container 6c436ba0c6855e2fe23e53f6f5609b89288d39036f364dd23803e2d99b9124e3: Status 404 returned error can't find the container with id 6c436ba0c6855e2fe23e53f6f5609b89288d39036f364dd23803e2d99b9124e3 Nov 01 22:30:01.046825 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:01.046798 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77baf41a_f5ea_46f0_a803_1c96e0e05cb4.slice/crio-f777ee6b243f6f7c8ea650c10de77db2f09ff462860b0b84a07311e1f1fa50e8 WatchSource:0}: Error finding container f777ee6b243f6f7c8ea650c10de77db2f09ff462860b0b84a07311e1f1fa50e8: Status 404 returned error can't find the container with id f777ee6b243f6f7c8ea650c10de77db2f09ff462860b0b84a07311e1f1fa50e8 Nov 01 22:30:01.047513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.047488 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:01.093507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.093461 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:01.128736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.128709 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:01.179460 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:01.179425 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d58e932_c410_4733_84dd_0ccfeee4f4ff.slice/crio-8a2ba89603fc3724f6f396f7734523ba17ebf4e1f0cb2d80c571240af7f4d524 WatchSource:0}: Error finding container 8a2ba89603fc3724f6f396f7734523ba17ebf4e1f0cb2d80c571240af7f4d524: Status 404 returned error can't find the container with id 8a2ba89603fc3724f6f396f7734523ba17ebf4e1f0cb2d80c571240af7f4d524 Nov 01 22:30:01.181264 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:01.181232 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fea932_ae37_4272_9bdb_00d95fbae449.slice/crio-c11c4309d0ea1c7d21eba6178c3e7f0251b8fdd2891943468bc024c1a180ccdb WatchSource:0}: Error finding container c11c4309d0ea1c7d21eba6178c3e7f0251b8fdd2891943468bc024c1a180ccdb: Status 404 returned error can't find the container with id c11c4309d0ea1c7d21eba6178c3e7f0251b8fdd2891943468bc024c1a180ccdb Nov 01 22:30:01.798095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.798063 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerStarted","Data":"dcd287cf875a2a96b754766fa5a844f6f41b815ce7aee981b0410c47d85b84cb"} Nov 01 22:30:01.798231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.798118 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerStarted","Data":"f777ee6b243f6f7c8ea650c10de77db2f09ff462860b0b84a07311e1f1fa50e8"} Nov 01 22:30:01.799628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.799603 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerStarted","Data":"9532a642ff95fe01e689c1842ae0a03cda7028089db25df22bea8aa4ec986f6a"} Nov 01 22:30:01.799738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.799643 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerStarted","Data":"8a2ba89603fc3724f6f396f7734523ba17ebf4e1f0cb2d80c571240af7f4d524"} Nov 01 22:30:01.801411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.801379 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerStarted","Data":"9b29ac635a854ced4bc43ec7aae9df78c7d85e78ed970a3550686f5ed1b360e8"} Nov 01 22:30:01.801533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.801435 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerStarted","Data":"649f18a15ca6053c9ea75525deea2399092420ab084bd9a8b2a301caeea283d4"} Nov 01 22:30:01.803171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.803139 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerStarted","Data":"449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881"} Nov 01 22:30:01.803292 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.803187 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerStarted","Data":"fa082f8edf0b19c876bbdf4e772471163b127e452ec064660b526ef0e1578bd6"} Nov 01 22:30:01.804813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.804791 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerStarted","Data":"b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3"} Nov 01 22:30:01.804920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.804833 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerStarted","Data":"6c436ba0c6855e2fe23e53f6f5609b89288d39036f364dd23803e2d99b9124e3"} Nov 01 22:30:01.806431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.806409 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerStarted","Data":"926f8b17fac290308cd5b1e106e1a2c86bad16446aae3f474df605a133357a22"} Nov 01 22:30:01.806534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.806445 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerStarted","Data":"c11c4309d0ea1c7d21eba6178c3e7f0251b8fdd2891943468bc024c1a180ccdb"} Nov 01 22:30:01.815840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.815800 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" podStartSLOduration=1.81578824 podStartE2EDuration="1.81578824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.814835144 +0000 UTC m=+2748.788737683" watchObservedRunningTime="2026-11-01 22:30:01.81578824 +0000 UTC m=+2748.789690780" Nov 01 22:30:01.833634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.833588 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" podStartSLOduration=1.833575137 podStartE2EDuration="1.833575137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.832509748 +0000 UTC m=+2748.806412287" watchObservedRunningTime="2026-11-01 22:30:01.833575137 +0000 UTC m=+2748.807477654" Nov 01 22:30:01.849975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.849930 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" podStartSLOduration=1.849917424 podStartE2EDuration="1.849917424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.849554129 +0000 UTC m=+2748.823456671" watchObservedRunningTime="2026-11-01 22:30:01.849917424 +0000 UTC m=+2748.823819963" Nov 01 22:30:01.870731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.870676 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" podStartSLOduration=1.870664427 podStartE2EDuration="1.870664427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.866904286 +0000 UTC m=+2748.840806826" watchObservedRunningTime="2026-11-01 22:30:01.870664427 +0000 UTC m=+2748.844566966" Nov 01 22:30:01.886524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.886469 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" podStartSLOduration=1.886454005 podStartE2EDuration="1.886454005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.884257984 +0000 UTC m=+2748.858160524" watchObservedRunningTime="2026-11-01 22:30:01.886454005 +0000 UTC m=+2748.860356583" Nov 01 22:30:01.902133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:01.902072 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" podStartSLOduration=1.902012136 podStartE2EDuration="1.902012136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.900933956 +0000 UTC m=+2748.874836494" watchObservedRunningTime="2026-11-01 22:30:01.902012136 +0000 UTC m=+2748.875914677" Nov 01 22:30:02.128408 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:02.128382 4395 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0cc2bb_16b1_43e9_b5d8_c18ca2dfdcae.slice/crio-conmon-b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3.scope/pids.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0cc2bb_16b1_43e9_b5d8_c18ca2dfdcae.slice/crio-conmon-b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3.scope/pids.max: no such device Nov 01 22:30:02.137897 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:02.137871 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454fdedb_8928_45dc_a238_d7c02814a6d4.slice/crio-449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:30:02.815205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.815171 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerID="9532a642ff95fe01e689c1842ae0a03cda7028089db25df22bea8aa4ec986f6a" exitCode=60 Nov 01 22:30:02.815703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.815261 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerDied","Data":"9532a642ff95fe01e689c1842ae0a03cda7028089db25df22bea8aa4ec986f6a"} Nov 01 22:30:02.815703 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.815573 4395 scope.go:122] "RemoveContainer" containerID="9532a642ff95fe01e689c1842ae0a03cda7028089db25df22bea8aa4ec986f6a" Nov 01 22:30:02.817077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.817050 4395 generic.go:356] "Generic (PLEG): container finished" podID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerID="9b29ac635a854ced4bc43ec7aae9df78c7d85e78ed970a3550686f5ed1b360e8" exitCode=60 Nov 01 22:30:02.817204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.817079 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerDied","Data":"9b29ac635a854ced4bc43ec7aae9df78c7d85e78ed970a3550686f5ed1b360e8"} Nov 01 22:30:02.817384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.817356 4395 scope.go:122] "RemoveContainer" containerID="9b29ac635a854ced4bc43ec7aae9df78c7d85e78ed970a3550686f5ed1b360e8" Nov 01 22:30:02.818873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.818849 4395 generic.go:356] "Generic (PLEG): container finished" podID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerID="449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881" exitCode=60 Nov 01 22:30:02.819050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.818924 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerDied","Data":"449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881"} Nov 01 22:30:02.819557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.819503 4395 scope.go:122] "RemoveContainer" containerID="449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881" Nov 01 22:30:02.821397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.821316 4395 generic.go:356] "Generic (PLEG): container finished" podID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerID="b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3" exitCode=60 Nov 01 22:30:02.821397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.821382 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerDied","Data":"b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3"} Nov 01 22:30:02.821610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.821594 4395 scope.go:122] "RemoveContainer" containerID="b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3" Nov 01 22:30:02.823132 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.823110 4395 generic.go:356] "Generic (PLEG): container finished" podID="61fea932-ae37-4272-9bdb-00d95fbae449" containerID="926f8b17fac290308cd5b1e106e1a2c86bad16446aae3f474df605a133357a22" exitCode=60 Nov 01 22:30:02.823230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.823191 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerDied","Data":"926f8b17fac290308cd5b1e106e1a2c86bad16446aae3f474df605a133357a22"} Nov 01 22:30:02.823392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.823373 4395 scope.go:122] "RemoveContainer" containerID="926f8b17fac290308cd5b1e106e1a2c86bad16446aae3f474df605a133357a22" Nov 01 22:30:02.824831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.824808 4395 generic.go:356] "Generic (PLEG): container finished" podID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerID="dcd287cf875a2a96b754766fa5a844f6f41b815ce7aee981b0410c47d85b84cb" exitCode=60 Nov 01 22:30:02.824938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.824845 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerDied","Data":"dcd287cf875a2a96b754766fa5a844f6f41b815ce7aee981b0410c47d85b84cb"} Nov 01 22:30:02.825078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:02.825064 4395 scope.go:122] "RemoveContainer" containerID="dcd287cf875a2a96b754766fa5a844f6f41b815ce7aee981b0410c47d85b84cb" Nov 01 22:30:03.834568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.834526 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerStarted","Data":"a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0"} Nov 01 22:30:03.836612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.836586 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerStarted","Data":"543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099"} Nov 01 22:30:03.838389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.838361 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerStarted","Data":"a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c"} Nov 01 22:30:03.840070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.840014 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerStarted","Data":"7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd"} Nov 01 22:30:03.841918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.841899 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerStarted","Data":"8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b"} Nov 01 22:30:03.843834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:03.843812 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerStarted","Data":"c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3"} Nov 01 22:30:04.824878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.824848 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:30:04.857210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.857174 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerID="a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0" exitCode=60 Nov 01 22:30:04.858131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.858098 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerDied","Data":"a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0"} Nov 01 22:30:04.858280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.858170 4395 scope.go:122] "RemoveContainer" containerID="9532a642ff95fe01e689c1842ae0a03cda7028089db25df22bea8aa4ec986f6a" Nov 01 22:30:04.858727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.858696 4395 scope.go:122] "RemoveContainer" containerID="a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0" Nov 01 22:30:04.859174 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.859145 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7_odh-ai-gateway-infra(6d58e932-c410-4733-84dd-0ccfeee4f4ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" Nov 01 22:30:04.864724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.864694 4395 generic.go:356] "Generic (PLEG): container finished" podID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerID="10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d" exitCode=0 Nov 01 22:30:04.864855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.864806 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" Nov 01 22:30:04.864913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.864890 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" event={"ID":"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13","Type":"ContainerDied","Data":"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d"} Nov 01 22:30:04.864967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.864939 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9" event={"ID":"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13","Type":"ContainerDied","Data":"4d4407dec445a68f0badc2cc904c82d880384ae3fdbc5685b7415e5eabcc36f1"} Nov 01 22:30:04.869038 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.867828 4395 generic.go:356] "Generic (PLEG): container finished" podID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerID="543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099" exitCode=60 Nov 01 22:30:04.869038 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.867926 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerDied","Data":"543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099"} Nov 01 22:30:04.869038 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.868406 4395 scope.go:122] "RemoveContainer" containerID="543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099" Nov 01 22:30:04.869038 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.868814 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp_odh-ai-gateway-infra(83889a6b-6f8b-4fb2-bd12-d4ac41ed6931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" Nov 01 22:30:04.870451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.870432 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.870663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.870648 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.870851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.870837 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.870992 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871244 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.871229 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.871380 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.871549 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.871695 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.871901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.871886 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\") pod \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\" (UID: \"e4fc2ab6-2f90-42e8-8ac8-2701917ddd13\") " Nov 01 22:30:04.872580 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.872321 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:04.873309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.873289 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:04.873586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.873570 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:04.874492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.874248 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:04.874492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.874464 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:04.882188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.877577 4395 generic.go:356] "Generic (PLEG): container finished" podID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerID="a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c" exitCode=60 Nov 01 22:30:04.882188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.877669 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerDied","Data":"a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c"} Nov 01 22:30:04.882188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.878159 4395 scope.go:122] "RemoveContainer" containerID="a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c" Nov 01 22:30:04.882188 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.878520 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd_odh-ai-gateway-infra(454fdedb-8928-45dc-a238-d7c02814a6d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" Nov 01 22:30:04.890931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.889452 4395 generic.go:356] "Generic (PLEG): container finished" podID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerID="7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd" exitCode=60 Nov 01 22:30:04.890931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.889535 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerDied","Data":"7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd"} Nov 01 22:30:04.890931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.889990 4395 scope.go:122] "RemoveContainer" containerID="7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd" Nov 01 22:30:04.890931 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.890445 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt_odh-ai-gateway-infra(cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" Nov 01 22:30:04.890931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.890716 4395 scope.go:122] "RemoveContainer" containerID="10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d" Nov 01 22:30:04.891265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.891237 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:04.898568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.898537 4395 generic.go:356] "Generic (PLEG): container finished" podID="61fea932-ae37-4272-9bdb-00d95fbae449" containerID="8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b" exitCode=60 Nov 01 22:30:04.898728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.898676 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerDied","Data":"8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b"} Nov 01 22:30:04.899176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.899157 4395 scope.go:122] "RemoveContainer" containerID="8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b" Nov 01 22:30:04.899570 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.899546 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9_odh-ai-gateway-infra(61fea932-ae37-4272-9bdb-00d95fbae449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" Nov 01 22:30:04.900886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.900658 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "kube-api-access-cbxpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:04.905062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.904519 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.923210 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.923239 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token" pod "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" (UID: "e4fc2ab6-2f90-42e8-8ac8-2701917ddd13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.925001 4395 generic.go:356] "Generic (PLEG): container finished" podID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerID="c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3" exitCode=60 Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.925176 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerDied","Data":"c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3"} Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.925666 4395 scope.go:122] "RemoveContainer" containerID="c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3" Nov 01 22:30:04.927211 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.926211 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx_odh-ai-gateway-infra(77baf41a-f5ea-46f0-a803-1c96e0e05cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" Nov 01 22:30:04.935885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.935835 4395 scope.go:122] "RemoveContainer" containerID="10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d" Nov 01 22:30:04.945864 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:04.945659 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d\": container with ID starting with 10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d not found: ID does not exist" containerID="10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d" Nov 01 22:30:04.945864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.945713 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d"} err="failed to get container status \"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d\": rpc error: code = NotFound desc = could not find container \"10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d\": container with ID starting with 10b974f6629443865f02ba8d504cab80948b0d3f98e460325e7bc4135d35fa8d not found: ID does not exist" Nov 01 22:30:04.945864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.945742 4395 scope.go:122] "RemoveContainer" containerID="9b29ac635a854ced4bc43ec7aae9df78c7d85e78ed970a3550686f5ed1b360e8" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984827 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984871 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984888 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984903 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxpt\" (UniqueName: \"kubernetes.io/projected/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-kube-api-access-cbxpt\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984918 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984933 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984948 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.026049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:04.984962 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:05.211285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.204639 4395 scope.go:122] "RemoveContainer" containerID="449345dc3f52f0faa5c0632b2829fb83cfba9fa017ce3286eba83c1a18aba881" Nov 01 22:30:05.234763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.234729 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:30:05.239881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.239854 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-grcn9"] Nov 01 22:30:05.239999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.239979 4395 scope.go:122] "RemoveContainer" containerID="b757a1401e4095f7be557912cd085c3cc79628460b8e209d0e6aeb9a7425d0f3" Nov 01 22:30:05.254247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.254223 4395 scope.go:122] "RemoveContainer" containerID="926f8b17fac290308cd5b1e106e1a2c86bad16446aae3f474df605a133357a22" Nov 01 22:30:05.268138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.268122 4395 scope.go:122] "RemoveContainer" containerID="dcd287cf875a2a96b754766fa5a844f6f41b815ce7aee981b0410c47d85b84cb" Nov 01 22:30:05.942309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.941047 4395 scope.go:122] "RemoveContainer" containerID="a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0" Nov 01 22:30:05.942309 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.941435 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7_odh-ai-gateway-infra(6d58e932-c410-4733-84dd-0ccfeee4f4ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" Nov 01 22:30:05.947152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.947122 4395 scope.go:122] "RemoveContainer" containerID="543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099" Nov 01 22:30:05.947445 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.947415 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp_odh-ai-gateway-infra(83889a6b-6f8b-4fb2-bd12-d4ac41ed6931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" Nov 01 22:30:05.949455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.949431 4395 scope.go:122] "RemoveContainer" containerID="a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c" Nov 01 22:30:05.949752 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.949731 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd_odh-ai-gateway-infra(454fdedb-8928-45dc-a238-d7c02814a6d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" Nov 01 22:30:05.951223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.951197 4395 scope.go:122] "RemoveContainer" containerID="7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd" Nov 01 22:30:05.951488 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.951466 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt_odh-ai-gateway-infra(cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" Nov 01 22:30:05.952944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.952923 4395 scope.go:122] "RemoveContainer" containerID="8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b" Nov 01 22:30:05.953230 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.953206 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9_odh-ai-gateway-infra(61fea932-ae37-4272-9bdb-00d95fbae449)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" Nov 01 22:30:05.954795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:05.954778 4395 scope.go:122] "RemoveContainer" containerID="c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3" Nov 01 22:30:05.955001 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:05.954980 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx_odh-ai-gateway-infra(77baf41a-f5ea-46f0-a803-1c96e0e05cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" Nov 01 22:30:07.166929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:07.166893 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" path="/var/lib/kubelet/pods/e4fc2ab6-2f90-42e8-8ac8-2701917ddd13/volumes" Nov 01 22:30:10.101128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:10.101093 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:30:11.383539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.383506 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:30:11.384207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.383740 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerName="maas-api" containerID="cri-o://3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50" gracePeriod=30 Nov 01 22:30:11.412331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.412292 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:11.756301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.756254 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:11.769209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.768975 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\" (UniqueName: \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\") pod \"454fdedb-8928-45dc-a238-d7c02814a6d4\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " Nov 01 22:30:11.769209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.769188 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\") pod \"454fdedb-8928-45dc-a238-d7c02814a6d4\" (UID: \"454fdedb-8928-45dc-a238-d7c02814a6d4\") " Nov 01 22:30:11.770432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.770378 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca" pod "454fdedb-8928-45dc-a238-d7c02814a6d4" (UID: "454fdedb-8928-45dc-a238-d7c02814a6d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:11.783469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.783386 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg" pod "454fdedb-8928-45dc-a238-d7c02814a6d4" (UID: "454fdedb-8928-45dc-a238-d7c02814a6d4"). InnerVolumeSpecName "kube-api-access-m6gxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:11.873295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.873258 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/454fdedb-8928-45dc-a238-d7c02814a6d4-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.873295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.873297 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gxg\" (UniqueName: \"kubernetes.io/projected/454fdedb-8928-45dc-a238-d7c02814a6d4-kube-api-access-m6gxg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:11.893656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.893629 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:30:11.999644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.999616 4395 generic.go:356] "Generic (PLEG): container finished" podID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerID="3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50" exitCode=0 Nov 01 22:30:11.999804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.999662 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" Nov 01 22:30:11.999804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.999702 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" event={"ID":"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32","Type":"ContainerDied","Data":"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50"} Nov 01 22:30:11.999804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.999772 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" event={"ID":"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32","Type":"ContainerDied","Data":"80f79f42675118a59faed5c4143a460575e876056b1ce97ab5c96ec197294a8f"} Nov 01 22:30:11.999941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:11.999806 4395 scope.go:122] "RemoveContainer" containerID="3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50" Nov 01 22:30:12.001174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.001151 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" Nov 01 22:30:12.001291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.001170 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd" event={"ID":"454fdedb-8928-45dc-a238-d7c02814a6d4","Type":"ContainerDied","Data":"fa082f8edf0b19c876bbdf4e772471163b127e452ec064660b526ef0e1578bd6"} Nov 01 22:30:12.011767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.011750 4395 scope.go:122] "RemoveContainer" containerID="3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50" Nov 01 22:30:12.012056 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:12.012009 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50\": container with ID starting with 3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50 not found: ID does not exist" containerID="3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50" Nov 01 22:30:12.012109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.012050 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50"} err="failed to get container status \"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50\": rpc error: code = NotFound desc = could not find container \"3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50\": container with ID starting with 3075dc418f68aa9c90b86e280097ff83bf67f71bfd7996c83d6d5fb02de11b50 not found: ID does not exist" Nov 01 22:30:12.012109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.012066 4395 scope.go:122] "RemoveContainer" containerID="a2dd15b47fc516001b07c73c8146d6a0ed0f8ee8453d833d240b174e1591e58c" Nov 01 22:30:12.033606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.033578 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:12.037572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.037551 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-51e3b7-29892870-trvwd"] Nov 01 22:30:12.075727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.075695 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\") pod \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " Nov 01 22:30:12.075727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.075726 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\") pod \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " Nov 01 22:30:12.075871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.075818 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\" (UniqueName: \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\") pod \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " Nov 01 22:30:12.075948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.075925 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\") pod \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\" (UID: \"2bc93f2f-a73a-4e3c-adc1-40e94cff4c32\") " Nov 01 22:30:12.076403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.076375 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca" pod "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" (UID: "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:12.076815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.076793 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:12.078015 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.077991 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs" pod "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" (UID: "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:12.078172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.078016 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls" pod "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" (UID: "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:12.078172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.078127 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp" pod "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" (UID: "2bc93f2f-a73a-4e3c-adc1-40e94cff4c32"). InnerVolumeSpecName "kube-api-access-55zpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:12.185054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.184319 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:12.185054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.184363 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:12.185054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.184379 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zpp\" (UniqueName: \"kubernetes.io/projected/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32-kube-api-access-55zpp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:12.354296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.354261 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:30:12.358965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:12.358937 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz"] Nov 01 22:30:13.166999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:13.166969 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" path="/var/lib/kubelet/pods/2bc93f2f-a73a-4e3c-adc1-40e94cff4c32/volumes" Nov 01 22:30:13.167490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:13.167472 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" path="/var/lib/kubelet/pods/454fdedb-8928-45dc-a238-d7c02814a6d4/volumes" Nov 01 22:30:14.649115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:14.649069 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-51e3b7-558d7c64bf-5jwgz" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.176:8443/health\": context deadline exceeded" Nov 01 22:30:15.101933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:15.101872 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" containerID="cri-o://618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b" gracePeriod=25 Nov 01 22:30:15.158273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:15.158242 4395 scope.go:122] "RemoveContainer" containerID="c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3" Nov 01 22:30:16.044593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:16.044565 4395 generic.go:356] "Generic (PLEG): container finished" podID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerID="5cf2591f27b8622bba1ec1f2664390829b6b310527da3e0f1773a22a53529121" exitCode=60 Nov 01 22:30:16.044967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:16.044600 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerDied","Data":"5cf2591f27b8622bba1ec1f2664390829b6b310527da3e0f1773a22a53529121"} Nov 01 22:30:16.044967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:16.044648 4395 scope.go:122] "RemoveContainer" containerID="c765bee715d00a1afb080a0dd6d66a8108b5a8178bf876e0ca688c2e411ff5a3" Nov 01 22:30:16.045113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:16.044969 4395 scope.go:122] "RemoveContainer" containerID="5cf2591f27b8622bba1ec1f2664390829b6b310527da3e0f1773a22a53529121" Nov 01 22:30:16.045324 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:16.045296 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx_odh-ai-gateway-infra(77baf41a-f5ea-46f0-a803-1c96e0e05cb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" Nov 01 22:30:16.505639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:16.505610 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:30:17.074658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.074631 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:17.161301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.158868 4395 scope.go:122] "RemoveContainer" containerID="7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd" Nov 01 22:30:17.251686 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.251663 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:17.310763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.309539 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\" (UniqueName: \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\") pod \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " Nov 01 22:30:17.310763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.309799 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\") pod \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\" (UID: \"77baf41a-f5ea-46f0-a803-1c96e0e05cb4\") " Nov 01 22:30:17.310763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.310219 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca" pod "77baf41a-f5ea-46f0-a803-1c96e0e05cb4" (UID: "77baf41a-f5ea-46f0-a803-1c96e0e05cb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:17.310763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.310697 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:17.312266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.312237 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7" pod "77baf41a-f5ea-46f0-a803-1c96e0e05cb4" (UID: "77baf41a-f5ea-46f0-a803-1c96e0e05cb4"). InnerVolumeSpecName "kube-api-access-kgkk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:17.411933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:17.411900 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkk7\" (UniqueName: \"kubernetes.io/projected/77baf41a-f5ea-46f0-a803-1c96e0e05cb4-kube-api-access-kgkk7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:18.064724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.064690 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerStarted","Data":"b552a5c04bc2ff3d2230dc969461ab778ef3a33e424368c7820dc026d978bc56"} Nov 01 22:30:18.066449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.066419 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" Nov 01 22:30:18.066578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.066417 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx" event={"ID":"77baf41a-f5ea-46f0-a803-1c96e0e05cb4","Type":"ContainerDied","Data":"f777ee6b243f6f7c8ea650c10de77db2f09ff462860b0b84a07311e1f1fa50e8"} Nov 01 22:30:18.066644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.066576 4395 scope.go:122] "RemoveContainer" containerID="5cf2591f27b8622bba1ec1f2664390829b6b310527da3e0f1773a22a53529121" Nov 01 22:30:18.108219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.108193 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:18.114374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.114347 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-53b784-29892870-j9wwx"] Nov 01 22:30:18.157858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.157830 4395 scope.go:122] "RemoveContainer" containerID="a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0" Nov 01 22:30:18.158020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.158003 4395 scope.go:122] "RemoveContainer" containerID="543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099" Nov 01 22:30:18.158356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.158334 4395 scope.go:122] "RemoveContainer" containerID="8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b" Nov 01 22:30:18.240084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.240057 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:30:18.241133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241111 4395 cpu_manager.go:414] "containerMap: removing container" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.241290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241275 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.241391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241380 4395 cpu_manager.go:414] "containerMap: removing container" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.241473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241461 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.241633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241620 4395 cpu_manager.go:414] "containerMap: removing container" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.241721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241711 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.241797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241789 4395 cpu_manager.go:414] "containerMap: removing container" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.241872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241862 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.241970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.241960 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerName="maas-api" Nov 01 22:30:18.242081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242058 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerName="maas-api" Nov 01 22:30:18.242203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242124 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" Nov 01 22:30:18.242203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242132 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" Nov 01 22:30:18.242341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242323 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc93f2f-a73a-4e3c-adc1-40e94cff4c32" containerName="maas-api" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242356 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fc2ab6-2f90-42e8-8ac8-2701917ddd13" containerName="istio-proxy" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242367 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242379 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242389 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242398 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:18.242407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.242408 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="454fdedb-8928-45dc-a238-d7c02814a6d4" containerName="cleanup" Nov 01 22:30:18.247988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.247968 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.254718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.254695 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:30:18.338128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.331073 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj97\" (UniqueName: \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.338128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.331219 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.362078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.339550 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60807f-dockercfg-pt7s5\"" Nov 01 22:30:18.362078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.339648 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60807f\"" Nov 01 22:30:18.443539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.439362 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj97\" (UniqueName: \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.443539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.439489 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.443539 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.440456 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.504403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.500159 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj97\" (UniqueName: \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\") pod \"payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.652731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.652705 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:18.939833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:18.939808 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:30:18.940539 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:18.940517 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a7b03_3fb2_4009_be54_e820e315fcaf.slice/crio-d5f639521b8d888e2f403a08536bcb4c6412fe6e1222fd0d93f95c8b7588d454 WatchSource:0}: Error finding container d5f639521b8d888e2f403a08536bcb4c6412fe6e1222fd0d93f95c8b7588d454: Status 404 returned error can't find the container with id d5f639521b8d888e2f403a08536bcb4c6412fe6e1222fd0d93f95c8b7588d454 Nov 01 22:30:19.077070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.076017 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerStarted","Data":"d873cdb6a4cd49faf3fbd6e316155951141a0998c961f54c6d1eab96406b6cbf"} Nov 01 22:30:19.079782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.079019 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerStarted","Data":"77a77d6a40ff28660d444e2de40122efcd1c92a19a4dfb6b101e7a6baf9725c8"} Nov 01 22:30:19.081596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.081567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" event={"ID":"c15a7b03-3fb2-4009-be54-e820e315fcaf","Type":"ContainerStarted","Data":"4e340e5835faca17e483bf78385c48500c67c2db2f56e0fb30ea1ddc6af484d5"} Nov 01 22:30:19.081725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.081621 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" event={"ID":"c15a7b03-3fb2-4009-be54-e820e315fcaf","Type":"ContainerStarted","Data":"d5f639521b8d888e2f403a08536bcb4c6412fe6e1222fd0d93f95c8b7588d454"} Nov 01 22:30:19.081725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.081693 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:19.084166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.084144 4395 generic.go:356] "Generic (PLEG): container finished" podID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerID="b552a5c04bc2ff3d2230dc969461ab778ef3a33e424368c7820dc026d978bc56" exitCode=60 Nov 01 22:30:19.084277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.084233 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerDied","Data":"b552a5c04bc2ff3d2230dc969461ab778ef3a33e424368c7820dc026d978bc56"} Nov 01 22:30:19.084339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.084322 4395 scope.go:122] "RemoveContainer" containerID="7803f158fac42e0f6e02279132e20380a3021c16e16c4a35141c82db7b263ecd" Nov 01 22:30:19.084701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.084673 4395 scope.go:122] "RemoveContainer" containerID="b552a5c04bc2ff3d2230dc969461ab778ef3a33e424368c7820dc026d978bc56" Nov 01 22:30:19.084958 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:19.084932 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt_odh-ai-gateway-infra(cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" Nov 01 22:30:19.087119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.087095 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerStarted","Data":"e96a46f655c5098266dee5366918f8d1ef6be0d763fbea1a1a9b8705988f789a"} Nov 01 22:30:19.098234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.098208 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:19.171991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.171946 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" path="/var/lib/kubelet/pods/77baf41a-f5ea-46f0-a803-1c96e0e05cb4/volumes" Nov 01 22:30:19.172563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:19.172249 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" podStartSLOduration=1.172233275 podStartE2EDuration="1.172233275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:19.16743593 +0000 UTC m=+2766.141338469" watchObservedRunningTime="2026-11-01 22:30:19.172233275 +0000 UTC m=+2766.146135815" Nov 01 22:30:20.137234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.137199 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:20.170759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.169160 4395 generic.go:356] "Generic (PLEG): container finished" podID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerID="d873cdb6a4cd49faf3fbd6e316155951141a0998c961f54c6d1eab96406b6cbf" exitCode=60 Nov 01 22:30:20.170759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.169340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerDied","Data":"d873cdb6a4cd49faf3fbd6e316155951141a0998c961f54c6d1eab96406b6cbf"} Nov 01 22:30:20.170759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.169393 4395 scope.go:122] "RemoveContainer" containerID="a4a1855bb71e730fa99c8ed3381536bb9418622e1cb206a64b82ad0c8757ecd0" Nov 01 22:30:20.190977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.190946 4395 generic.go:356] "Generic (PLEG): container finished" podID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerID="77a77d6a40ff28660d444e2de40122efcd1c92a19a4dfb6b101e7a6baf9725c8" exitCode=60 Nov 01 22:30:20.191795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.191699 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerDied","Data":"77a77d6a40ff28660d444e2de40122efcd1c92a19a4dfb6b101e7a6baf9725c8"} Nov 01 22:30:20.196345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.196300 4395 scope.go:122] "RemoveContainer" containerID="77a77d6a40ff28660d444e2de40122efcd1c92a19a4dfb6b101e7a6baf9725c8" Nov 01 22:30:20.200552 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:20.200521 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp_odh-ai-gateway-infra(83889a6b-6f8b-4fb2-bd12-d4ac41ed6931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" Nov 01 22:30:20.210066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.207866 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:20.263626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.261387 4395 generic.go:356] "Generic (PLEG): container finished" podID="61fea932-ae37-4272-9bdb-00d95fbae449" containerID="e96a46f655c5098266dee5366918f8d1ef6be0d763fbea1a1a9b8705988f789a" exitCode=60 Nov 01 22:30:20.263626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.262503 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerDied","Data":"e96a46f655c5098266dee5366918f8d1ef6be0d763fbea1a1a9b8705988f789a"} Nov 01 22:30:20.281164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.276151 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:20.327296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.325761 4395 scope.go:122] "RemoveContainer" containerID="543ecb18779bd16d0a3d62ed5ee03a7f51139c4471841d0e8b3adebda1993099" Nov 01 22:30:20.371523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.371392 4395 scope.go:122] "RemoveContainer" containerID="8209d6378e762dd7c5198db37b950723e5549b7d59e355282bde4dc563abab6b" Nov 01 22:30:20.619585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.619246 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:20.711595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.711516 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\" (UniqueName: \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\") pod \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " Nov 01 22:30:20.715269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.715224 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9" pod "cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" (UID: "cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae"). InnerVolumeSpecName "kube-api-access-chpv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:20.728935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.722112 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:20.750616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.750595 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:20.814082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.814049 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\") pod \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\" (UID: \"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae\") " Nov 01 22:30:20.814489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.814458 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca" pod "cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" (UID: "cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.815201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.815178 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.815361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.815345 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpv9\" (UniqueName: \"kubernetes.io/projected/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae-kube-api-access-chpv9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.915957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.915928 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\") pod \"61fea932-ae37-4272-9bdb-00d95fbae449\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " Nov 01 22:30:20.916137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.915975 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\") pod \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " Nov 01 22:30:20.916137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916068 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\") pod \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\" (UID: \"6d58e932-c410-4733-84dd-0ccfeee4f4ff\") " Nov 01 22:30:20.916137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916106 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\" (UniqueName: \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\") pod \"61fea932-ae37-4272-9bdb-00d95fbae449\" (UID: \"61fea932-ae37-4272-9bdb-00d95fbae449\") " Nov 01 22:30:20.916321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916300 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca" pod "61fea932-ae37-4272-9bdb-00d95fbae449" (UID: "61fea932-ae37-4272-9bdb-00d95fbae449"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.916482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916454 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca" pod "6d58e932-c410-4733-84dd-0ccfeee4f4ff" (UID: "6d58e932-c410-4733-84dd-0ccfeee4f4ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.916793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916772 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fea932-ae37-4272-9bdb-00d95fbae449-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.916895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.916797 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d58e932-c410-4733-84dd-0ccfeee4f4ff-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:20.918406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.918387 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5" pod "6d58e932-c410-4733-84dd-0ccfeee4f4ff" (UID: "6d58e932-c410-4733-84dd-0ccfeee4f4ff"). InnerVolumeSpecName "kube-api-access-k9ps5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:20.918482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:20.918450 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv" pod "61fea932-ae37-4272-9bdb-00d95fbae449" (UID: "61fea932-ae37-4272-9bdb-00d95fbae449"). InnerVolumeSpecName "kube-api-access-2vtlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:21.018238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.018202 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ps5\" (UniqueName: \"kubernetes.io/projected/6d58e932-c410-4733-84dd-0ccfeee4f4ff-kube-api-access-k9ps5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.018238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.018238 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtlv\" (UniqueName: \"kubernetes.io/projected/61fea932-ae37-4272-9bdb-00d95fbae449-kube-api-access-2vtlv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.271344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.271320 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" Nov 01 22:30:21.271742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.271310 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt" event={"ID":"cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae","Type":"ContainerDied","Data":"6c436ba0c6855e2fe23e53f6f5609b89288d39036f364dd23803e2d99b9124e3"} Nov 01 22:30:21.271742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.271449 4395 scope.go:122] "RemoveContainer" containerID="b552a5c04bc2ff3d2230dc969461ab778ef3a33e424368c7820dc026d978bc56" Nov 01 22:30:21.272888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.272870 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" Nov 01 22:30:21.272997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.272947 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9" event={"ID":"61fea932-ae37-4272-9bdb-00d95fbae449","Type":"ContainerDied","Data":"c11c4309d0ea1c7d21eba6178c3e7f0251b8fdd2891943468bc024c1a180ccdb"} Nov 01 22:30:21.289608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.289561 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" event={"ID":"6d58e932-c410-4733-84dd-0ccfeee4f4ff","Type":"ContainerDied","Data":"8a2ba89603fc3724f6f396f7734523ba17ebf4e1f0cb2d80c571240af7f4d524"} Nov 01 22:30:21.289966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.289933 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7" Nov 01 22:30:21.313540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.313506 4395 scope.go:122] "RemoveContainer" containerID="e96a46f655c5098266dee5366918f8d1ef6be0d763fbea1a1a9b8705988f789a" Nov 01 22:30:21.341843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.341809 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:21.346678 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.346655 4395 scope.go:122] "RemoveContainer" containerID="d873cdb6a4cd49faf3fbd6e316155951141a0998c961f54c6d1eab96406b6cbf" Nov 01 22:30:21.349845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.349814 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c6bb56-29892870-5ggrt"] Nov 01 22:30:21.360913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.360888 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:21.368044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.368008 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0def36-29892870-2glx9"] Nov 01 22:30:21.403050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.401080 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:21.417415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.417138 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-f758d9-29892870-nxjr7"] Nov 01 22:30:21.531931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.531018 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" containerID="cri-o://c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545" gracePeriod=25 Nov 01 22:30:21.605707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.605670 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:21.645786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.645751 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\") pod \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " Nov 01 22:30:21.645973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.645912 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\" (UniqueName: \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\") pod \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\" (UID: \"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931\") " Nov 01 22:30:21.646139 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.646114 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca" pod "83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" (UID: "83889a6b-6f8b-4fb2-bd12-d4ac41ed6931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:21.646652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.646630 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-maas-api-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.663769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.648234 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb" pod "83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" (UID: "83889a6b-6f8b-4fb2-bd12-d4ac41ed6931"). InnerVolumeSpecName "kube-api-access-mlnsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:21.748357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:21.748321 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnsb\" (UniqueName: \"kubernetes.io/projected/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931-kube-api-access-mlnsb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:22.107612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.107566 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:22.358326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.357203 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" event={"ID":"83889a6b-6f8b-4fb2-bd12-d4ac41ed6931","Type":"ContainerDied","Data":"649f18a15ca6053c9ea75525deea2399092420ab084bd9a8b2a301caeea283d4"} Nov 01 22:30:22.358326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.357297 4395 scope.go:122] "RemoveContainer" containerID="77a77d6a40ff28660d444e2de40122efcd1c92a19a4dfb6b101e7a6baf9725c8" Nov 01 22:30:22.358326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.357399 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp" Nov 01 22:30:22.403627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.403595 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:22.404581 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:22.404552 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:30:22.407214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:22.407194 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-09656e-29892870-zd4xp"] Nov 01 22:30:23.232202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:23.232172 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" path="/var/lib/kubelet/pods/61fea932-ae37-4272-9bdb-00d95fbae449/volumes" Nov 01 22:30:23.233007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:23.232988 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" path="/var/lib/kubelet/pods/6d58e932-c410-4733-84dd-0ccfeee4f4ff/volumes" Nov 01 22:30:23.233690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:23.233674 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" path="/var/lib/kubelet/pods/83889a6b-6f8b-4fb2-bd12-d4ac41ed6931/volumes" Nov 01 22:30:23.234541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:23.234527 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" path="/var/lib/kubelet/pods/cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae/volumes" Nov 01 22:30:25.271821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:25.271787 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 22:30:25.289078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:25.289046 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 22:30:29.468447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.468409 4395 generic.go:356] "Generic (PLEG): container finished" podID="cefce3b9-edf9-402c-a781-2ad03e534677" containerID="563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372" exitCode=0 Nov 01 22:30:29.468951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.468479 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerDied","Data":"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372"} Nov 01 22:30:29.468951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.468905 4395 scope.go:122] "RemoveContainer" containerID="563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372" Nov 01 22:30:29.470553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.470520 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:29.470649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.470593 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:29.470649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:29.470623 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:30:30.265097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:30.265069 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:30:30.500054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:30.499863 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerStarted","Data":"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737"} Nov 01 22:30:30.520268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:30.520190 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:30:30.521776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:30.521751 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:30.521904 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:30.521793 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:31.092784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:31.092731 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:31.520976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:31.520943 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:31.521454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:31.520993 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:32.107538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:32.107502 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:32.520780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:32.520749 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:32.520990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:32.520811 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:33.529053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:33.528398 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:33.529053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:33.528465 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:34.335671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:34.335633 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:30:34.520319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:34.520283 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:34.520495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:34.520336 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:35.454764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.454324 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:30:35.472364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.472337 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\") pod \"09a0e765-4dad-4a0a-ae48-9b629641ed23\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " Nov 01 22:30:35.472783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.472761 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\" (UniqueName: \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\") pod \"09a0e765-4dad-4a0a-ae48-9b629641ed23\" (UID: \"09a0e765-4dad-4a0a-ae48-9b629641ed23\") " Nov 01 22:30:35.473145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.472785 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume" pod "09a0e765-4dad-4a0a-ae48-9b629641ed23" (UID: "09a0e765-4dad-4a0a-ae48-9b629641ed23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:35.473886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.473858 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09a0e765-4dad-4a0a-ae48-9b629641ed23-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:35.510192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.510156 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz" pod "09a0e765-4dad-4a0a-ae48-9b629641ed23" (UID: "09a0e765-4dad-4a0a-ae48-9b629641ed23"). InnerVolumeSpecName "kube-api-access-tddsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:35.520701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.520669 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 01 22:30:35.520909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.520740 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 01 22:30:35.551955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.551924 4395 generic.go:356] "Generic (PLEG): container finished" podID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerID="618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b" exitCode=0 Nov 01 22:30:35.552087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.551976 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" Nov 01 22:30:35.552087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.552014 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" event={"ID":"09a0e765-4dad-4a0a-ae48-9b629641ed23","Type":"ContainerDied","Data":"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b"} Nov 01 22:30:35.552168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.552098 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm" event={"ID":"09a0e765-4dad-4a0a-ae48-9b629641ed23","Type":"ContainerDied","Data":"6244b126f868acab76ccdf09792b2a9939cc33d9eb2dabfd4fa20ff761846b41"} Nov 01 22:30:35.552168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.552123 4395 scope.go:122] "RemoveContainer" containerID="618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b" Nov 01 22:30:35.563671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.563653 4395 scope.go:122] "RemoveContainer" containerID="618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b" Nov 01 22:30:35.563925 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:35.563907 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b\": container with ID starting with 618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b not found: ID does not exist" containerID="618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b" Nov 01 22:30:35.563970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.563933 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b"} err="failed to get container status \"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b\": rpc error: code = NotFound desc = could not find container \"618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b\": container with ID starting with 618f13af5e73182f74ac8a027c2104b8e3abf7f58c1ed6206e11ecb07c65041b not found: ID does not exist" Nov 01 22:30:35.575519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.575491 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddsz\" (UniqueName: \"kubernetes.io/projected/09a0e765-4dad-4a0a-ae48-9b629641ed23-kube-api-access-tddsz\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:35.584420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.584396 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:30:35.590563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:35.590543 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-51e3b7-769bbdbc5d-7zxkm"] Nov 01 22:30:36.161405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.161370 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt"] Nov 01 22:30:36.162335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162307 4395 cpu_manager.go:414] "containerMap: removing container" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" Nov 01 22:30:36.162335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162333 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162353 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162360 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162374 4395 cpu_manager.go:414] "containerMap: removing container" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162381 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162392 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162399 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162409 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162417 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162430 4395 cpu_manager.go:414] "containerMap: removing container" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162436 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162469 4395 cpu_manager.go:414] "containerMap: removing container" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162475 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162484 4395 cpu_manager.go:414] "containerMap: removing container" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162489 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162510 4395 cpu_manager.go:414] "containerMap: removing container" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162517 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162528 4395 cpu_manager.go:414] "containerMap: removing container" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162534 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77baf41a-f5ea-46f0-a803-1c96e0e05cb4" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162548 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.162546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162554 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162563 4395 cpu_manager.go:414] "containerMap: removing container" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162569 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162730 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162745 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.162758 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163845 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163864 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163876 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163885 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" containerName="payload-processing" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163895 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163907 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163916 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:36.164429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.163926 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:36.169262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.169244 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.189093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.177547 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:30:36.189093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.177590 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:36.189093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.179706 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt"] Nov 01 22:30:36.287523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.287488 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.288245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.288215 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.288464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.288434 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4lj\" (UniqueName: \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.390265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.390228 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.390428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.390388 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4lj\" (UniqueName: \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.390500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.390484 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.390909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.390891 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.393068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.393009 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.402146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.402122 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4lj\" (UniqueName: \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\") pod \"maas-api-revoke-sub-5c95f262-hwjvt\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.493949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.493844 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:36.527656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.527625 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:30:36.528983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.528961 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:30:36.813014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:36.812986 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt"] Nov 01 22:30:36.813441 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:36.813420 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46f8a42_4b1a_4f1a_ad7c_765a9be08c3d.slice/crio-20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432 WatchSource:0}: Error finding container 20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432: Status 404 returned error can't find the container with id 20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432 Nov 01 22:30:37.166533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:37.166501 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a0e765-4dad-4a0a-ae48-9b629641ed23" path="/var/lib/kubelet/pods/09a0e765-4dad-4a0a-ae48-9b629641ed23/volumes" Nov 01 22:30:37.572097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:37.572005 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" event={"ID":"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d","Type":"ContainerStarted","Data":"b474681855f969b7daef1a158f1ed9bc0e772dd9f1301f9eed19b6890ae5c36c"} Nov 01 22:30:37.572548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:37.572164 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" event={"ID":"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d","Type":"ContainerStarted","Data":"20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432"} Nov 01 22:30:37.593525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:37.593480 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" podStartSLOduration=1.593465974 podStartE2EDuration="1.593465974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:37.589307996 +0000 UTC m=+2784.563210536" watchObservedRunningTime="2026-11-01 22:30:37.593465974 +0000 UTC m=+2784.567368513" Nov 01 22:30:38.580537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:38.580502 4395 generic.go:356] "Generic (PLEG): container finished" podID="b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" containerID="b474681855f969b7daef1a158f1ed9bc0e772dd9f1301f9eed19b6890ae5c36c" exitCode=0 Nov 01 22:30:38.580937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:38.580558 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" event={"ID":"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d","Type":"ContainerDied","Data":"b474681855f969b7daef1a158f1ed9bc0e772dd9f1301f9eed19b6890ae5c36c"} Nov 01 22:30:39.764251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.764221 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:39.899964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.899897 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\") pod \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " Nov 01 22:30:39.900276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.899982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\" (UniqueName: \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\") pod \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " Nov 01 22:30:39.900276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.900086 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\") pod \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\" (UID: \"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d\") " Nov 01 22:30:39.900421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.900311 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca" pod "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" (UID: "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:39.900688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.900665 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:39.902387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.902353 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca" pod "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" (UID: "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:39.902479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:39.902390 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj" pod "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" (UID: "b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d"). InnerVolumeSpecName "kube-api-access-xc4lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:40.002397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.002371 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc4lj\" (UniqueName: \"kubernetes.io/projected/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-kube-api-access-xc4lj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.002397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.002397 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.296112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.296073 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" containerID="cri-o://eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab" gracePeriod=45 Nov 01 22:30:40.318252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.318193 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" containerID="cri-o://9780100779d3f7cad1e6565a6d94eec35e39aba551e0aa19a5cf334266145ee6" gracePeriod=45 Nov 01 22:30:40.327806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.327778 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused" Nov 01 22:30:40.578556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.578532 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:30:40.600654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.600482 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" event={"ID":"b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d","Type":"ContainerDied","Data":"20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432"} Nov 01 22:30:40.600654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.600544 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20ee25fc9575f9d98beb8ec8998ab5dbbdbe279c8775bc0300718e5bc6cf3432" Nov 01 22:30:40.604047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.600620 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt" Nov 01 22:30:40.606737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.606435 4395 generic.go:356] "Generic (PLEG): container finished" podID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerID="9780100779d3f7cad1e6565a6d94eec35e39aba551e0aa19a5cf334266145ee6" exitCode=0 Nov 01 22:30:40.606737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.606659 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" event={"ID":"5fe2733d-471f-43d6-b75e-bf652ca28621","Type":"ContainerDied","Data":"9780100779d3f7cad1e6565a6d94eec35e39aba551e0aa19a5cf334266145ee6"} Nov 01 22:30:40.609741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.609132 4395 generic.go:356] "Generic (PLEG): container finished" podID="e59111f7-0000-4ee7-9647-1c9571db616b" containerID="eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab" exitCode=0 Nov 01 22:30:40.609741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.609191 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" event={"ID":"e59111f7-0000-4ee7-9647-1c9571db616b","Type":"ContainerDied","Data":"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab"} Nov 01 22:30:40.609741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.609222 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" event={"ID":"e59111f7-0000-4ee7-9647-1c9571db616b","Type":"ContainerDied","Data":"69af22712ada0ff262d3b573c09bee2a313f24d7da9eb1c8651f93aea15f2d87"} Nov 01 22:30:40.609741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.609244 4395 scope.go:122] "RemoveContainer" containerID="eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab" Nov 01 22:30:40.609741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.609378 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml" Nov 01 22:30:40.628207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.628179 4395 scope.go:122] "RemoveContainer" containerID="eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab" Nov 01 22:30:40.629804 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:40.629711 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab\": container with ID starting with eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab not found: ID does not exist" containerID="eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab" Nov 01 22:30:40.629804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.629751 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab"} err="failed to get container status \"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab\": rpc error: code = NotFound desc = could not find container \"eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab\": container with ID starting with eee6955a9e466108d9856fcc40c20f041f0008ee30c66ee8d92f2ce0e72b2aab not found: ID does not exist" Nov 01 22:30:40.640920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.640880 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:30:40.714123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714093 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.714327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714134 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.714327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714246 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.714465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\" (UniqueName: \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.714465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714411 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.714465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714421 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.714623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.714499 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\") pod \"e59111f7-0000-4ee7-9647-1c9571db616b\" (UID: \"e59111f7-0000-4ee7-9647-1c9571db616b\") " Nov 01 22:30:40.715088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.715065 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.718076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.715315 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.718076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.715528 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.718076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.717856 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:40.718076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.718004 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "kube-api-access-65qsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:40.718393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.718379 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm" pod "e59111f7-0000-4ee7-9647-1c9571db616b" (UID: "e59111f7-0000-4ee7-9647-1c9571db616b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.816404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816371 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.816906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816510 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\" (UniqueName: \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.816906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816553 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.816906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816644 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.816906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816715 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.816906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816754 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\") pod \"5fe2733d-471f-43d6-b75e-bf652ca28621\" (UID: \"5fe2733d-471f-43d6-b75e-bf652ca28621\") " Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.816941 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817348 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817363 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e59111f7-0000-4ee7-9647-1c9571db616b-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817373 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817381 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qsx\" (UniqueName: \"kubernetes.io/projected/e59111f7-0000-4ee7-9647-1c9571db616b-kube-api-access-65qsx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817373 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817391 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817400 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e59111f7-0000-4ee7-9647-1c9571db616b-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.817450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.817449 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.818820 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.818795 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:40.818998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.818980 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "kube-api-access-t4ntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:40.819253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.819231 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm" pod "5fe2733d-471f-43d6-b75e-bf652ca28621" (UID: "5fe2733d-471f-43d6-b75e-bf652ca28621"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:30:40.918553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.918526 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.918553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.918555 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.918553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.918569 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe2733d-471f-43d6-b75e-bf652ca28621-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.918839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.918584 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ntg\" (UniqueName: \"kubernetes.io/projected/5fe2733d-471f-43d6-b75e-bf652ca28621-kube-api-access-t4ntg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.918839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.918598 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe2733d-471f-43d6-b75e-bf652ca28621-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:40.936221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.936192 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 22:30:40.939596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:40.939570 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-2-w1-53b784-kserve-88f8d7576-2vnml"] Nov 01 22:30:41.091455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.091333 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:30:41.168294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.168266 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" path="/var/lib/kubelet/pods/e59111f7-0000-4ee7-9647-1c9571db616b/volumes" Nov 01 22:30:41.613876 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.613835 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:30:41.625684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.625646 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" event={"ID":"5fe2733d-471f-43d6-b75e-bf652ca28621","Type":"ContainerDied","Data":"307189e74ba154e0e93c4c268e3b706f5c364d1851a77e1229911c1a92997903"} Nov 01 22:30:41.625979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.625951 4395 scope.go:122] "RemoveContainer" containerID="9780100779d3f7cad1e6565a6d94eec35e39aba551e0aa19a5cf334266145ee6" Nov 01 22:30:41.626229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.626202 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6" Nov 01 22:30:41.682325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.682246 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 22:30:41.700783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.700704 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/premium-w1-53b784-kserve-98b787848-tzpc6"] Nov 01 22:30:41.979548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:41.979515 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:30:42.106371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.098204 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\") pod \"fefe4562-493a-42b1-91b5-9a237a4426bd\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " Nov 01 22:30:42.106371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.098378 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\" (UniqueName: \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\") pod \"fefe4562-493a-42b1-91b5-9a237a4426bd\" (UID: \"fefe4562-493a-42b1-91b5-9a237a4426bd\") " Nov 01 22:30:42.111466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.111405 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume" pod "fefe4562-493a-42b1-91b5-9a237a4426bd" (UID: "fefe4562-493a-42b1-91b5-9a237a4426bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:42.241330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.236196 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefe4562-493a-42b1-91b5-9a237a4426bd-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:42.276275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.276223 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp" pod "fefe4562-493a-42b1-91b5-9a237a4426bd" (UID: "fefe4562-493a-42b1-91b5-9a237a4426bd"). InnerVolumeSpecName "kube-api-access-x6ldp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:42.362408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.362317 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ldp\" (UniqueName: \"kubernetes.io/projected/fefe4562-493a-42b1-91b5-9a237a4426bd-kube-api-access-x6ldp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:42.633225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.633126 4395 generic.go:356] "Generic (PLEG): container finished" podID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerID="c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545" exitCode=0 Nov 01 22:30:42.633225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.633181 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" Nov 01 22:30:42.633225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.633218 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" event={"ID":"fefe4562-493a-42b1-91b5-9a237a4426bd","Type":"ContainerDied","Data":"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545"} Nov 01 22:30:42.633511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.633264 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt" event={"ID":"fefe4562-493a-42b1-91b5-9a237a4426bd","Type":"ContainerDied","Data":"444f9866f19ffa51b5359acfd7f764db6e4281a510b2336f6c132cae3b56e70a"} Nov 01 22:30:42.633511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.633291 4395 scope.go:122] "RemoveContainer" containerID="c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545" Nov 01 22:30:42.644618 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.644602 4395 scope.go:122] "RemoveContainer" containerID="c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545" Nov 01 22:30:42.644896 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:42.644875 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545\": container with ID starting with c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545 not found: ID does not exist" containerID="c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545" Nov 01 22:30:42.644946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.644906 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545"} err="failed to get container status \"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545\": rpc error: code = NotFound desc = could not find container \"c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545\": container with ID starting with c989a0b7b4a8dac2998164aacf18a2bbf309ed867d8614b6734cb1d749018545 not found: ID does not exist" Nov 01 22:30:42.661830 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.661802 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:30:42.665266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:42.665242 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-f758d9-554fcc4d84-gg7tt"] Nov 01 22:30:43.166436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:43.166407 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" path="/var/lib/kubelet/pods/5fe2733d-471f-43d6-b75e-bf652ca28621/volumes" Nov 01 22:30:43.166825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:43.166811 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" path="/var/lib/kubelet/pods/fefe4562-493a-42b1-91b5-9a237a4426bd/volumes" Nov 01 22:30:46.722429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:46.722389 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:30:46.723084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:46.722669 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" podUID="1a487d68-697d-4d12-bd59-1ce78c41e084" containerName="maas-api" containerID="cri-o://1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b" gracePeriod=30 Nov 01 22:30:47.200194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.200170 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:30:47.269430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.269402 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\") pod \"1a487d68-697d-4d12-bd59-1ce78c41e084\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " Nov 01 22:30:47.269625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.269567 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\") pod \"1a487d68-697d-4d12-bd59-1ce78c41e084\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " Nov 01 22:30:47.269625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.269614 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\") pod \"1a487d68-697d-4d12-bd59-1ce78c41e084\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " Nov 01 22:30:47.269774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.269660 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\" (UniqueName: \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\") pod \"1a487d68-697d-4d12-bd59-1ce78c41e084\" (UID: \"1a487d68-697d-4d12-bd59-1ce78c41e084\") " Nov 01 22:30:47.270002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.269971 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca" pod "1a487d68-697d-4d12-bd59-1ce78c41e084" (UID: "1a487d68-697d-4d12-bd59-1ce78c41e084"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:47.270570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.270547 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a487d68-697d-4d12-bd59-1ce78c41e084-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:47.271754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.271731 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs" pod "1a487d68-697d-4d12-bd59-1ce78c41e084" (UID: "1a487d68-697d-4d12-bd59-1ce78c41e084"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:47.271921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.271900 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls" pod "1a487d68-697d-4d12-bd59-1ce78c41e084" (UID: "1a487d68-697d-4d12-bd59-1ce78c41e084"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:47.271921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.271913 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv" pod "1a487d68-697d-4d12-bd59-1ce78c41e084" (UID: "1a487d68-697d-4d12-bd59-1ce78c41e084"). InnerVolumeSpecName "kube-api-access-ck6cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:47.373944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.372573 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:47.373944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.372612 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6cv\" (UniqueName: \"kubernetes.io/projected/1a487d68-697d-4d12-bd59-1ce78c41e084-kube-api-access-ck6cv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:47.373944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.372629 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a487d68-697d-4d12-bd59-1ce78c41e084-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:47.696356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.696321 4395 generic.go:356] "Generic (PLEG): container finished" podID="1a487d68-697d-4d12-bd59-1ce78c41e084" containerID="1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b" exitCode=0 Nov 01 22:30:47.696552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.696375 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" Nov 01 22:30:47.696552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.696385 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" event={"ID":"1a487d68-697d-4d12-bd59-1ce78c41e084","Type":"ContainerDied","Data":"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b"} Nov 01 22:30:47.696552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.696477 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl" event={"ID":"1a487d68-697d-4d12-bd59-1ce78c41e084","Type":"ContainerDied","Data":"2bffe4251eaa01d20f7d942a2e9c5d7f53c63b687fc80e83b22cd45cd3568f47"} Nov 01 22:30:47.696552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.696510 4395 scope.go:122] "RemoveContainer" containerID="1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b" Nov 01 22:30:47.708655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.708639 4395 scope.go:122] "RemoveContainer" containerID="1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b" Nov 01 22:30:47.708934 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:47.708909 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b\": container with ID starting with 1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b not found: ID does not exist" containerID="1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b" Nov 01 22:30:47.709002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.708946 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b"} err="failed to get container status \"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b\": rpc error: code = NotFound desc = could not find container \"1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b\": container with ID starting with 1c5c9400c65e883454ef1f368fbc9b44d18ace796ad067fd32412106717f232b not found: ID does not exist" Nov 01 22:30:47.722043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.722000 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:30:47.725872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:47.725853 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2c5135-5b95fcb78f-z8dpl"] Nov 01 22:30:48.108655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:48.108627 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:30:48.329584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:48.329559 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:30:49.167486 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:49.167430 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a487d68-697d-4d12-bd59-1ce78c41e084" path="/var/lib/kubelet/pods/1a487d68-697d-4d12-bd59-1ce78c41e084/volumes" Nov 01 22:30:50.352239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.348047 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:30:50.352239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.348330 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" podUID="6b55ddbc-379c-412b-a308-5162592a305d" containerName="maas-api" containerID="cri-o://83b836b3b75eeb673d334346ce4fd308fd5a323b535bf80fcd366885b8b18b05" gracePeriod=30 Nov 01 22:30:50.413967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.413934 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:30:50.414236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.414215 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" podUID="33a1ac87-4847-4614-81ff-df3f3e8f258e" containerName="payload-processing" containerID="cri-o://519a25bf03ff225662e6d6eac8adde205b3de8207b183d50e79e78c3a4221796" gracePeriod=30 Nov 01 22:30:50.425235 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.425192 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:30:50.427095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.425501 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" podUID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" containerName="payload-pre-processing" containerID="cri-o://dc0b65bafd2dade7ed7ff69355daaa8c6b0862254f390aacbf159822360d50fe" gracePeriod=30 Nov 01 22:30:50.755054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.751397 4395 generic.go:356] "Generic (PLEG): container finished" podID="6b55ddbc-379c-412b-a308-5162592a305d" containerID="83b836b3b75eeb673d334346ce4fd308fd5a323b535bf80fcd366885b8b18b05" exitCode=0 Nov 01 22:30:50.755054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.751702 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" event={"ID":"6b55ddbc-379c-412b-a308-5162592a305d","Type":"ContainerDied","Data":"83b836b3b75eeb673d334346ce4fd308fd5a323b535bf80fcd366885b8b18b05"} Nov 01 22:30:50.755054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.754220 4395 generic.go:356] "Generic (PLEG): container finished" podID="33a1ac87-4847-4614-81ff-df3f3e8f258e" containerID="519a25bf03ff225662e6d6eac8adde205b3de8207b183d50e79e78c3a4221796" exitCode=0 Nov 01 22:30:50.755054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.754295 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" event={"ID":"33a1ac87-4847-4614-81ff-df3f3e8f258e","Type":"ContainerDied","Data":"519a25bf03ff225662e6d6eac8adde205b3de8207b183d50e79e78c3a4221796"} Nov 01 22:30:50.762901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.756674 4395 generic.go:356] "Generic (PLEG): container finished" podID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" containerID="dc0b65bafd2dade7ed7ff69355daaa8c6b0862254f390aacbf159822360d50fe" exitCode=0 Nov 01 22:30:50.762901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.756743 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" event={"ID":"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc","Type":"ContainerDied","Data":"dc0b65bafd2dade7ed7ff69355daaa8c6b0862254f390aacbf159822360d50fe"} Nov 01 22:30:50.968202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:50.959384 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:30:51.041941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.041912 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\") pod \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " Nov 01 22:30:51.042673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.042288 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\" (UniqueName: \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\") pod \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\" (UID: \"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc\") " Nov 01 22:30:51.042673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.042488 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume" pod "4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" (UID: "4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:51.042894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.042829 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.044981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.044951 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n" pod "4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" (UID: "4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc"). InnerVolumeSpecName "kube-api-access-6hn8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:51.053765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.053692 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:30:51.113052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.110611 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:30:51.144277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.144245 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\") pod \"6b55ddbc-379c-412b-a308-5162592a305d\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " Nov 01 22:30:51.144487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.144315 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\") pod \"6b55ddbc-379c-412b-a308-5162592a305d\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " Nov 01 22:30:51.144487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.144354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\") pod \"6b55ddbc-379c-412b-a308-5162592a305d\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " Nov 01 22:30:51.144487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.144388 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\" (UniqueName: \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\") pod \"6b55ddbc-379c-412b-a308-5162592a305d\" (UID: \"6b55ddbc-379c-412b-a308-5162592a305d\") " Nov 01 22:30:51.145072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.144723 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca" pod "6b55ddbc-379c-412b-a308-5162592a305d" (UID: "6b55ddbc-379c-412b-a308-5162592a305d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:51.145443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.145129 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hn8n\" (UniqueName: \"kubernetes.io/projected/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc-kube-api-access-6hn8n\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.145443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.145155 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b55ddbc-379c-412b-a308-5162592a305d-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.147600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.147573 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm" pod "6b55ddbc-379c-412b-a308-5162592a305d" (UID: "6b55ddbc-379c-412b-a308-5162592a305d"). InnerVolumeSpecName "kube-api-access-hnffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:51.149384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.149363 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs" pod "6b55ddbc-379c-412b-a308-5162592a305d" (UID: "6b55ddbc-379c-412b-a308-5162592a305d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:51.149769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.149745 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls" pod "6b55ddbc-379c-412b-a308-5162592a305d" (UID: "6b55ddbc-379c-412b-a308-5162592a305d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:51.245902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.245863 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\" (UniqueName: \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\") pod \"33a1ac87-4847-4614-81ff-df3f3e8f258e\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " Nov 01 22:30:51.246137 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.245982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\") pod \"33a1ac87-4847-4614-81ff-df3f3e8f258e\" (UID: \"33a1ac87-4847-4614-81ff-df3f3e8f258e\") " Nov 01 22:30:51.246516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.246480 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume" pod "33a1ac87-4847-4614-81ff-df3f3e8f258e" (UID: "33a1ac87-4847-4614-81ff-df3f3e8f258e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:51.246837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.246787 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.246837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.246811 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33a1ac87-4847-4614-81ff-df3f3e8f258e-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.246837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.246827 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b55ddbc-379c-412b-a308-5162592a305d-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.247050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.246849 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnffm\" (UniqueName: \"kubernetes.io/projected/6b55ddbc-379c-412b-a308-5162592a305d-kube-api-access-hnffm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.248410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.248381 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57" pod "33a1ac87-4847-4614-81ff-df3f3e8f258e" (UID: "33a1ac87-4847-4614-81ff-df3f3e8f258e"). InnerVolumeSpecName "kube-api-access-vld57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:51.348427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.348357 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vld57\" (UniqueName: \"kubernetes.io/projected/33a1ac87-4847-4614-81ff-df3f3e8f258e-kube-api-access-vld57\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.765148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.765119 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" Nov 01 22:30:51.765586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.765113 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8" event={"ID":"33a1ac87-4847-4614-81ff-df3f3e8f258e","Type":"ContainerDied","Data":"3288e4e7948c74ae9349330802e24ed6c6560cbf42fbf95779950a5b0bca58fb"} Nov 01 22:30:51.765586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.765267 4395 scope.go:122] "RemoveContainer" containerID="519a25bf03ff225662e6d6eac8adde205b3de8207b183d50e79e78c3a4221796" Nov 01 22:30:51.766692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.766664 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" Nov 01 22:30:51.766839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.766662 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn" event={"ID":"4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc","Type":"ContainerDied","Data":"5c406a6beb82ad17e573a748deb4118c2a4943a4e95b85e7a7fe354d4b01e36d"} Nov 01 22:30:51.768202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.768128 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" Nov 01 22:30:51.768202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.768158 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg" event={"ID":"6b55ddbc-379c-412b-a308-5162592a305d","Type":"ContainerDied","Data":"7630b8dbbc318d8d250a09ac880fa2591b531d02c123f7e55d1555de01bcfc42"} Nov 01 22:30:51.779940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.779168 4395 scope.go:122] "RemoveContainer" containerID="dc0b65bafd2dade7ed7ff69355daaa8c6b0862254f390aacbf159822360d50fe" Nov 01 22:30:51.788840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.788818 4395 scope.go:122] "RemoveContainer" containerID="83b836b3b75eeb673d334346ce4fd308fd5a323b535bf80fcd366885b8b18b05" Nov 01 22:30:51.795927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.795902 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:30:51.801929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.801905 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6bb56-67c55ff778-46fdg"] Nov 01 22:30:51.816670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.816650 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:30:51.819933 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.819913 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6bb56-749c9ddc86-kb8p8"] Nov 01 22:30:51.828924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.828902 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:30:51.833692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:51.833673 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6bb56-76dc475c55-r59sn"] Nov 01 22:30:52.302125 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:52.302085 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:30:52.442898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.442861 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 22:30:52.443602 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.443568 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" podUID="8914d34d-6ae3-4767-b974-e3674b1212eb" containerName="maas-api" containerID="cri-o://93af37a4cfe9385718180c16dabc5d900d918a6c6c02aefd853ab0ca5d71698c" gracePeriod=30 Nov 01 22:30:52.807067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.806989 4395 generic.go:356] "Generic (PLEG): container finished" podID="8914d34d-6ae3-4767-b974-e3674b1212eb" containerID="93af37a4cfe9385718180c16dabc5d900d918a6c6c02aefd853ab0ca5d71698c" exitCode=0 Nov 01 22:30:52.807494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.807196 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" event={"ID":"8914d34d-6ae3-4767-b974-e3674b1212eb","Type":"ContainerDied","Data":"93af37a4cfe9385718180c16dabc5d900d918a6c6c02aefd853ab0ca5d71698c"} Nov 01 22:30:52.878593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.878568 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 22:30:52.894611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.894582 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\") pod \"8914d34d-6ae3-4767-b974-e3674b1212eb\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " Nov 01 22:30:52.895495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.894767 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\") pod \"8914d34d-6ae3-4767-b974-e3674b1212eb\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " Nov 01 22:30:52.895495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.894974 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\" (UniqueName: \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\") pod \"8914d34d-6ae3-4767-b974-e3674b1212eb\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " Nov 01 22:30:52.895495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.895066 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca" pod "8914d34d-6ae3-4767-b974-e3674b1212eb" (UID: "8914d34d-6ae3-4767-b974-e3674b1212eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:52.895495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.895092 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\") pod \"8914d34d-6ae3-4767-b974-e3674b1212eb\" (UID: \"8914d34d-6ae3-4767-b974-e3674b1212eb\") " Nov 01 22:30:52.895744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.895718 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8914d34d-6ae3-4767-b974-e3674b1212eb-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:52.897774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.897752 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb" pod "8914d34d-6ae3-4767-b974-e3674b1212eb" (UID: "8914d34d-6ae3-4767-b974-e3674b1212eb"). InnerVolumeSpecName "kube-api-access-mg2rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:52.898074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.898050 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs" pod "8914d34d-6ae3-4767-b974-e3674b1212eb" (UID: "8914d34d-6ae3-4767-b974-e3674b1212eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:52.898450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.898422 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls" pod "8914d34d-6ae3-4767-b974-e3674b1212eb" (UID: "8914d34d-6ae3-4767-b974-e3674b1212eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:52.996892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.996861 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:52.996892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.996889 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8914d34d-6ae3-4767-b974-e3674b1212eb-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:52.996892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:52.996899 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg2rb\" (UniqueName: \"kubernetes.io/projected/8914d34d-6ae3-4767-b974-e3674b1212eb-kube-api-access-mg2rb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:53.170541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.170452 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a1ac87-4847-4614-81ff-df3f3e8f258e" path="/var/lib/kubelet/pods/33a1ac87-4847-4614-81ff-df3f3e8f258e/volumes" Nov 01 22:30:53.171148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.171123 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" path="/var/lib/kubelet/pods/4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc/volumes" Nov 01 22:30:53.171760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.171740 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b55ddbc-379c-412b-a308-5162592a305d" path="/var/lib/kubelet/pods/6b55ddbc-379c-412b-a308-5162592a305d/volumes" Nov 01 22:30:53.820346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.820311 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" Nov 01 22:30:53.820346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.820333 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l" event={"ID":"8914d34d-6ae3-4767-b974-e3674b1212eb","Type":"ContainerDied","Data":"29661c95f3d9583835ea9afa5b1cf48682d855008c1b604cc8a4d26fe4c6aa6d"} Nov 01 22:30:53.820901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.820426 4395 scope.go:122] "RemoveContainer" containerID="93af37a4cfe9385718180c16dabc5d900d918a6c6c02aefd853ab0ca5d71698c" Nov 01 22:30:53.841018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.840995 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 22:30:53.844938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:53.844914 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-53b784-589599c67-2jt4l"] Nov 01 22:30:54.605143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:54.605109 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:30:54.605389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:54.605362 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" containerID="cri-o://636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9" gracePeriod=30 Nov 01 22:30:55.167468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.167435 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8914d34d-6ae3-4767-b974-e3674b1212eb" path="/var/lib/kubelet/pods/8914d34d-6ae3-4767-b974-e3674b1212eb/volumes" Nov 01 22:30:55.213526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.213495 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh"] Nov 01 22:30:55.214091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214077 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214092 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214108 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214113 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214124 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8914d34d-6ae3-4767-b974-e3674b1212eb" containerName="maas-api" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214129 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8914d34d-6ae3-4767-b974-e3674b1212eb" containerName="maas-api" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214146 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" containerName="payload-pre-processing" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214151 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" containerName="payload-pre-processing" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214160 4395 cpu_manager.go:414] "containerMap: removing container" podUID="33a1ac87-4847-4614-81ff-df3f3e8f258e" containerName="payload-processing" Nov 01 22:30:55.214164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214164 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a1ac87-4847-4614-81ff-df3f3e8f258e" containerName="payload-processing" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214173 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214178 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d58e932-c410-4733-84dd-0ccfeee4f4ff" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214185 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214190 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0cc2bb-16b1-43e9-b5d8-c18ca2dfdcae" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214199 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1a487d68-697d-4d12-bd59-1ce78c41e084" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214203 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a487d68-697d-4d12-bd59-1ce78c41e084" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214211 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6b55ddbc-379c-412b-a308-5162592a305d" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214215 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b55ddbc-379c-412b-a308-5162592a305d" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214222 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214227 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214236 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" containerName="revoke-keys" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214241 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" containerName="revoke-keys" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214325 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" containerName="revoke-keys" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214334 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fea932-ae37-4272-9bdb-00d95fbae449" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214343 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="83889a6b-6f8b-4fb2-bd12-d4ac41ed6931" containerName="cleanup" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214348 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe2733d-471f-43d6-b75e-bf652ca28621" containerName="main" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214355 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59111f7-0000-4ee7-9647-1c9571db616b" containerName="main" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214360 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8914d34d-6ae3-4767-b974-e3674b1212eb" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214367 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4613ff0c-b459-47ec-94b6-5c8ef1c3b5fc" containerName="payload-pre-processing" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214373 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a1ac87-4847-4614-81ff-df3f3e8f258e" containerName="payload-processing" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214379 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a487d68-697d-4d12-bd59-1ce78c41e084" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214384 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b55ddbc-379c-412b-a308-5162592a305d" containerName="maas-api" Nov 01 22:30:55.214459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.214390 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe4562-493a-42b1-91b5-9a237a4426bd" containerName="payload-processing" Nov 01 22:30:55.219163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.219139 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.223733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.223565 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:30:55.224771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.224744 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:30:55.230283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.230259 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh"] Nov 01 22:30:55.365189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.365156 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.365364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.365215 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.365451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.365431 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zkml\" (UniqueName: \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.466736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.466650 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.466736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.466721 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.466983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.466823 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zkml\" (UniqueName: \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.467472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.467447 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.469458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.469435 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.479506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.479480 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zkml\" (UniqueName: \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\") pod \"maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.534682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.534655 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:55.820239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.820204 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh"] Nov 01 22:30:55.822544 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:55.822517 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac09e354_e878_47b4_b771_98d0cf316203.slice/crio-acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454 WatchSource:0}: Error finding container acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454: Status 404 returned error can't find the container with id acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454 Nov 01 22:30:55.870540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:55.870512 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" event={"ID":"ac09e354-e878-47b4-b771-98d0cf316203","Type":"ContainerStarted","Data":"acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454"} Nov 01 22:30:56.879555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:56.879521 4395 generic.go:356] "Generic (PLEG): container finished" podID="ac09e354-e878-47b4-b771-98d0cf316203" containerID="9365fd494a926262389d2ce951aaecd698a0af2d035a1cd8c6f0fba54d1a96d0" exitCode=0 Nov 01 22:30:56.879973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:56.879616 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" event={"ID":"ac09e354-e878-47b4-b771-98d0cf316203","Type":"ContainerDied","Data":"9365fd494a926262389d2ce951aaecd698a0af2d035a1cd8c6f0fba54d1a96d0"} Nov 01 22:30:57.910008 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:57.909976 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f"] Nov 01 22:30:57.914297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:57.914280 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:57.925217 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:57.925188 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f"] Nov 01 22:30:57.970795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:57.970760 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:30:57.971342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:57.971313 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" containerID="cri-o://dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349" gracePeriod=30 Nov 01 22:30:58.002504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.002476 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.002657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.002548 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.002657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.002633 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfk5\" (UniqueName: \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.133086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.112228 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfk5\" (UniqueName: \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.133086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.112535 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.133086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.112617 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.152630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.145420 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.152630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.151218 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.182367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.169310 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfk5\" (UniqueName: \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\") pod \"maas-api-revoke-sub-ed035fcf-lj56f\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.223550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.223501 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:58.231976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.231946 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:30:58.316070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.315993 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\" (UniqueName: \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\") pod \"ac09e354-e878-47b4-b771-98d0cf316203\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " Nov 01 22:30:58.316198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.316085 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\") pod \"ac09e354-e878-47b4-b771-98d0cf316203\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " Nov 01 22:30:58.316198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.316117 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\") pod \"ac09e354-e878-47b4-b771-98d0cf316203\" (UID: \"ac09e354-e878-47b4-b771-98d0cf316203\") " Nov 01 22:30:58.316652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.316623 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca" pod "ac09e354-e878-47b4-b771-98d0cf316203" (UID: "ac09e354-e878-47b4-b771-98d0cf316203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:58.316934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.316916 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:58.319090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.319064 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca" pod "ac09e354-e878-47b4-b771-98d0cf316203" (UID: "ac09e354-e878-47b4-b771-98d0cf316203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:58.324779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.324745 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml" pod "ac09e354-e878-47b4-b771-98d0cf316203" (UID: "ac09e354-e878-47b4-b771-98d0cf316203"). InnerVolumeSpecName "kube-api-access-5zkml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:58.418493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.418405 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac09e354-e878-47b4-b771-98d0cf316203-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:58.418493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.418450 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zkml\" (UniqueName: \"kubernetes.io/projected/ac09e354-e878-47b4-b771-98d0cf316203-kube-api-access-5zkml\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:58.446647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.446613 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j"] Nov 01 22:30:58.447347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.447329 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ac09e354-e878-47b4-b771-98d0cf316203" containerName="revoke-keys" Nov 01 22:30:58.447414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.447350 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac09e354-e878-47b4-b771-98d0cf316203" containerName="revoke-keys" Nov 01 22:30:58.447485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.447475 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac09e354-e878-47b4-b771-98d0cf316203" containerName="revoke-keys" Nov 01 22:30:58.452327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.452307 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.460222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.460196 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j"] Nov 01 22:30:58.519729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.519692 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.519920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.519897 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.520389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.520357 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5dp\" (UniqueName: \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.524816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.524794 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f"] Nov 01 22:30:58.526660 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:58.526639 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b035eaf_86fc_42bc_86fa_3d4def81625a.slice/crio-7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209 WatchSource:0}: Error finding container 7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209: Status 404 returned error can't find the container with id 7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209 Nov 01 22:30:58.624321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.624294 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5dp\" (UniqueName: \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.624476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.624351 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.624476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.624437 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.625065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.625013 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.626832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.626811 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.636229 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.636207 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5dp\" (UniqueName: \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\") pod \"maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.772181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.772155 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:30:58.907109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.906632 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" event={"ID":"6b035eaf-86fc-42bc-86fa-3d4def81625a","Type":"ContainerStarted","Data":"7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209"} Nov 01 22:30:58.911071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.908834 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" Nov 01 22:30:58.911071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.908931 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh" event={"ID":"ac09e354-e878-47b4-b771-98d0cf316203","Type":"ContainerDied","Data":"acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454"} Nov 01 22:30:58.911071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:58.908970 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acaf81cc4bc0741a9bf209f18368b8bf7ef8943066a62e3d735f5bdcd7d4c454" Nov 01 22:30:59.102259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.102231 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j"] Nov 01 22:30:59.103407 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:30:59.103383 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba8ee61_175d_488f_8022_9f782d68a8fa.slice/crio-30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb WatchSource:0}: Error finding container 30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb: Status 404 returned error can't find the container with id 30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb Nov 01 22:30:59.903445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.903421 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:30:59.916607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.916581 4395 generic.go:356] "Generic (PLEG): container finished" podID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerID="636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9" exitCode=0 Nov 01 22:30:59.916940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.916630 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" Nov 01 22:30:59.916940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.916663 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerDied","Data":"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9"} Nov 01 22:30:59.916940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.916716 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw" event={"ID":"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e","Type":"ContainerDied","Data":"fe55b44b08e9fb46a39d00c2d19893da82652165fdcad44f81cdf8a642e3d125"} Nov 01 22:30:59.916940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.916738 4395 scope.go:122] "RemoveContainer" containerID="636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9" Nov 01 22:30:59.918312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.918291 4395 generic.go:356] "Generic (PLEG): container finished" podID="6b035eaf-86fc-42bc-86fa-3d4def81625a" containerID="2c5a5338d53dc18d8ca04c87223545468f18223075872979a12c63e4bb7e4ef5" exitCode=0 Nov 01 22:30:59.918420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.918362 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" event={"ID":"6b035eaf-86fc-42bc-86fa-3d4def81625a","Type":"ContainerDied","Data":"2c5a5338d53dc18d8ca04c87223545468f18223075872979a12c63e4bb7e4ef5"} Nov 01 22:30:59.919931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.919894 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" event={"ID":"8ba8ee61-175d-488f-8022-9f782d68a8fa","Type":"ContainerStarted","Data":"c3ad621c19a1759df28244e3bcb47a2e4364d62b8f4879685dd4a3b9e0fb29ec"} Nov 01 22:30:59.920050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.919962 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" event={"ID":"8ba8ee61-175d-488f-8022-9f782d68a8fa","Type":"ContainerStarted","Data":"30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb"} Nov 01 22:30:59.927889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.927872 4395 scope.go:122] "RemoveContainer" containerID="87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38" Nov 01 22:30:59.939193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.939172 4395 scope.go:122] "RemoveContainer" containerID="636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9" Nov 01 22:30:59.939501 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:59.939475 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9\": container with ID starting with 636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9 not found: ID does not exist" containerID="636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9" Nov 01 22:30:59.939562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.939504 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9"} err="failed to get container status \"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9\": rpc error: code = NotFound desc = could not find container \"636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9\": container with ID starting with 636c5081c7e37c46565f6ff7693674af9f76c5dedfe95387240e7e68cc63eed9 not found: ID does not exist" Nov 01 22:30:59.939562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.939528 4395 scope.go:122] "RemoveContainer" containerID="87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38" Nov 01 22:30:59.939793 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:30:59.939776 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38\": container with ID starting with 87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38 not found: ID does not exist" containerID="87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38" Nov 01 22:30:59.939858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.939799 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38"} err="failed to get container status \"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38\": rpc error: code = NotFound desc = could not find container \"87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38\": container with ID starting with 87d4f456554be48a7eb14458f920c7b867ec22e36c40e77ba37b6cb98a493b38 not found: ID does not exist" Nov 01 22:30:59.971355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:30:59.971297 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" podStartSLOduration=1.971282976 podStartE2EDuration="1.971282976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:59.968524498 +0000 UTC m=+2806.942427037" watchObservedRunningTime="2026-11-01 22:30:59.971282976 +0000 UTC m=+2806.945185514" Nov 01 22:31:00.054277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054248 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054324 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054443 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054472 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054530 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054608 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054606 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:00.054809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054667 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054718 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054754 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\") pod \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\" (UID: \"4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e\") " Nov 01 22:31:00.054809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054755 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:00.055044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.054884 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:00.055232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055112 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:00.055634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055609 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:00.055795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055776 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.055863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055801 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.055863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055816 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.055863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055830 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.055863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.055845 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.056920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.056897 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "kube-api-access-cqc2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:00.057066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.056986 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:00.057156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.057096 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:00.057219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.057192 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy" pod "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" (UID: "4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:00.157355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.157279 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqc2n\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-kube-api-access-cqc2n\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.157355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.157311 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.157355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.157322 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.157355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.157331 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.301702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.301670 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:31:00.335850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.335817 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2c5135-openshift-default-7d55d56c89-8zbqw"] Nov 01 22:31:00.939743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.939706 4395 generic.go:356] "Generic (PLEG): container finished" podID="8ba8ee61-175d-488f-8022-9f782d68a8fa" containerID="c3ad621c19a1759df28244e3bcb47a2e4364d62b8f4879685dd4a3b9e0fb29ec" exitCode=0 Nov 01 22:31:00.940251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:00.939792 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" event={"ID":"8ba8ee61-175d-488f-8022-9f782d68a8fa","Type":"ContainerDied","Data":"c3ad621c19a1759df28244e3bcb47a2e4364d62b8f4879685dd4a3b9e0fb29ec"} Nov 01 22:31:01.121303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.121281 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:31:01.168478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.168443 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" path="/var/lib/kubelet/pods/4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e/volumes" Nov 01 22:31:01.280526 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.280492 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\" (UniqueName: \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\") pod \"6b035eaf-86fc-42bc-86fa-3d4def81625a\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " Nov 01 22:31:01.280728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.280565 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\") pod \"6b035eaf-86fc-42bc-86fa-3d4def81625a\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " Nov 01 22:31:01.280808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.280779 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\") pod \"6b035eaf-86fc-42bc-86fa-3d4def81625a\" (UID: \"6b035eaf-86fc-42bc-86fa-3d4def81625a\") " Nov 01 22:31:01.281225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.281196 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca" pod "6b035eaf-86fc-42bc-86fa-3d4def81625a" (UID: "6b035eaf-86fc-42bc-86fa-3d4def81625a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:01.281456 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.281437 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:01.283400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.283375 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca" pod "6b035eaf-86fc-42bc-86fa-3d4def81625a" (UID: "6b035eaf-86fc-42bc-86fa-3d4def81625a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:01.283515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.283465 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5" pod "6b035eaf-86fc-42bc-86fa-3d4def81625a" (UID: "6b035eaf-86fc-42bc-86fa-3d4def81625a"). InnerVolumeSpecName "kube-api-access-bwfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:01.382716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.382681 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwfk5\" (UniqueName: \"kubernetes.io/projected/6b035eaf-86fc-42bc-86fa-3d4def81625a-kube-api-access-bwfk5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:01.382716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.382716 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b035eaf-86fc-42bc-86fa-3d4def81625a-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:01.951331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.951302 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" Nov 01 22:31:01.951743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.951340 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f" event={"ID":"6b035eaf-86fc-42bc-86fa-3d4def81625a","Type":"ContainerDied","Data":"7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209"} Nov 01 22:31:01.951743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:01.951406 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2d8767126ad8c20230d370eaf31006490d3d69958f2211e6eeae4b57362209" Nov 01 22:31:02.144558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.144533 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:31:02.316793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.316758 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\") pod \"8ba8ee61-175d-488f-8022-9f782d68a8fa\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " Nov 01 22:31:02.317142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.316998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\") pod \"8ba8ee61-175d-488f-8022-9f782d68a8fa\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " Nov 01 22:31:02.317142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.317115 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\" (UniqueName: \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\") pod \"8ba8ee61-175d-488f-8022-9f782d68a8fa\" (UID: \"8ba8ee61-175d-488f-8022-9f782d68a8fa\") " Nov 01 22:31:02.317304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.317166 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca" pod "8ba8ee61-175d-488f-8022-9f782d68a8fa" (UID: "8ba8ee61-175d-488f-8022-9f782d68a8fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:02.318299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.318273 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:02.324071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.319894 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp" pod "8ba8ee61-175d-488f-8022-9f782d68a8fa" (UID: "8ba8ee61-175d-488f-8022-9f782d68a8fa"). InnerVolumeSpecName "kube-api-access-kg5dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:02.324528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.324502 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca" pod "8ba8ee61-175d-488f-8022-9f782d68a8fa" (UID: "8ba8ee61-175d-488f-8022-9f782d68a8fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:02.419873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.419836 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba8ee61-175d-488f-8022-9f782d68a8fa-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:02.419873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.419875 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5dp\" (UniqueName: \"kubernetes.io/projected/8ba8ee61-175d-488f-8022-9f782d68a8fa-kube-api-access-kg5dp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:02.636961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.634745 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:31:02.636961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.634815 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:31:02.959857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.959783 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" Nov 01 22:31:02.960261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.959781 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j" event={"ID":"8ba8ee61-175d-488f-8022-9f782d68a8fa","Type":"ContainerDied","Data":"30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb"} Nov 01 22:31:02.960261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:02.959907 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f0f3dd89a19c409388967cfbfcbb343591a2972d41a4ae5b9f26d4bc5593cb" Nov 01 22:31:03.337555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.337524 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:31:03.439136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439096 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439171 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439280 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439316 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439356 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439395 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439421 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439481 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439511 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\") pod \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\" (UID: \"7c2460d1-cb80-4a1a-9904-0eb21984fdc0\") " Nov 01 22:31:03.439927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.439786 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:03.440316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.440254 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.440628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.440575 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:03.440800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.440771 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:03.440921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.440904 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:03.441771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.441749 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:03.443199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.443178 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:03.444451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.444418 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "kube-api-access-nvbsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:03.444638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.444612 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:03.445872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.445846 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy" pod "7c2460d1-cb80-4a1a-9904-0eb21984fdc0" (UID: "7c2460d1-cb80-4a1a-9904-0eb21984fdc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:03.541365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541339 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbsh\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-kube-api-access-nvbsh\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541371 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541383 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541392 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541402 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541410 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541418 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.541494 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.541426 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2460d1-cb80-4a1a-9904-0eb21984fdc0-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:03.968048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.968004 4395 generic.go:356] "Generic (PLEG): container finished" podID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerID="dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349" exitCode=0 Nov 01 22:31:03.968528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.968084 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerDied","Data":"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349"} Nov 01 22:31:03.968528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.968087 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" Nov 01 22:31:03.968528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.968125 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7" event={"ID":"7c2460d1-cb80-4a1a-9904-0eb21984fdc0","Type":"ContainerDied","Data":"d4a2ae5e1ae5524b25dd1d0129016e74a58be1632667d6dfb6ddbf6bc3f2b897"} Nov 01 22:31:03.968528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.968147 4395 scope.go:122] "RemoveContainer" containerID="dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349" Nov 01 22:31:03.979065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.979044 4395 scope.go:122] "RemoveContainer" containerID="3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5" Nov 01 22:31:03.989875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.989859 4395 scope.go:122] "RemoveContainer" containerID="dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349" Nov 01 22:31:03.990172 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:03.990151 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349\": container with ID starting with dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349 not found: ID does not exist" containerID="dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349" Nov 01 22:31:03.990223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.990181 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349"} err="failed to get container status \"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349\": rpc error: code = NotFound desc = could not find container \"dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349\": container with ID starting with dd71274d2f9accbf73d22abcd2b96d74ad16de6134e40c592e36c59d8ef0d349 not found: ID does not exist" Nov 01 22:31:03.990223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.990199 4395 scope.go:122] "RemoveContainer" containerID="3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5" Nov 01 22:31:03.990469 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:03.990450 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5\": container with ID starting with 3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5 not found: ID does not exist" containerID="3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5" Nov 01 22:31:03.990507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:03.990474 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5"} err="failed to get container status \"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5\": rpc error: code = NotFound desc = could not find container \"3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5\": container with ID starting with 3ec33526a6e7f879c2997360ba4e9950c1cb5b5fb64867b87c2d9e57105ed9c5 not found: ID does not exist" Nov 01 22:31:04.000364 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.000340 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:31:04.003769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.003750 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c6bb56-openshift-default-54d9fcd6f9-8rfh7"] Nov 01 22:31:04.131629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.131589 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw"] Nov 01 22:31:04.132313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132293 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6b035eaf-86fc-42bc-86fa-3d4def81625a" containerName="revoke-keys" Nov 01 22:31:04.132313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132312 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b035eaf-86fc-42bc-86fa-3d4def81625a" containerName="revoke-keys" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132329 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132338 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132358 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132363 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132389 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8ba8ee61-175d-488f-8022-9f782d68a8fa" containerName="revoke-keys" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132393 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba8ee61-175d-488f-8022-9f782d68a8fa" containerName="revoke-keys" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132407 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132411 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:04.132498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132498 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:04.132997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132510 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:04.132997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132517 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba8ee61-175d-488f-8022-9f782d68a8fa" containerName="revoke-keys" Nov 01 22:31:04.132997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132525 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b035eaf-86fc-42bc-86fa-3d4def81625a" containerName="revoke-keys" Nov 01 22:31:04.132997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.132532 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:04.137649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.137628 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.141582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.141558 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:04.141707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.141660 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:04.146184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.146155 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw"] Nov 01 22:31:04.149868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.149839 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.150010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.149905 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.150489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.150116 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrmw\" (UniqueName: \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.251151 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.251121 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrmw\" (UniqueName: \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.251318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.251221 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.251318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.251259 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.251878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.251857 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.253845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.253823 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.271895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.271869 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrmw\" (UniqueName: \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\") pod \"maas-api-revoke-sub-2cd5e8d4-xv7fw\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.460397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.460366 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:04.747108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.747072 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw"] Nov 01 22:31:04.748184 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:04.748154 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02353b4a_6603_4af8_8f49_161b455410c9.slice/crio-ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08 WatchSource:0}: Error finding container ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08: Status 404 returned error can't find the container with id ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08 Nov 01 22:31:04.977558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:04.977509 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" event={"ID":"02353b4a-6603-4af8-8f49-161b455410c9","Type":"ContainerStarted","Data":"ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08"} Nov 01 22:31:05.165888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.165845 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" path="/var/lib/kubelet/pods/7c2460d1-cb80-4a1a-9904-0eb21984fdc0/volumes" Nov 01 22:31:05.951529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.951492 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h"] Nov 01 22:31:05.952303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.952279 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:05.952303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.952304 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2460d1-cb80-4a1a-9904-0eb21984fdc0" containerName="istio-proxy" Nov 01 22:31:05.952531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.952420 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db55ac9-8dc1-44c2-8d45-8b6ae5fcc27e" containerName="istio-proxy" Nov 01 22:31:05.956083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.956063 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:05.964880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.964855 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h"] Nov 01 22:31:05.995101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.993053 4395 generic.go:356] "Generic (PLEG): container finished" podID="02353b4a-6603-4af8-8f49-161b455410c9" containerID="7cbea1bfbc500151387d7a03bbb873f4992841fdf4587e2b5c70782c17195f58" exitCode=0 Nov 01 22:31:05.995101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:05.993163 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" event={"ID":"02353b4a-6603-4af8-8f49-161b455410c9","Type":"ContainerDied","Data":"7cbea1bfbc500151387d7a03bbb873f4992841fdf4587e2b5c70782c17195f58"} Nov 01 22:31:06.004604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.004576 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgjb\" (UniqueName: \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.004778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.004750 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.004967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.004949 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.105878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.105795 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.106064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.105908 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgjb\" (UniqueName: \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.106064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.106008 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.106547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.106515 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.108563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.108544 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.117940 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.117913 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgjb\" (UniqueName: \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\") pod \"maas-api-revoke-sub-9af18be7-mzr4h\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.277496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.277450 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:06.565790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:06.565423 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h"] Nov 01 22:31:06.566163 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:06.566132 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca5bc60_4778_4bca_be96_3ff971cde016.slice/crio-3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b WatchSource:0}: Error finding container 3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b: Status 404 returned error can't find the container with id 3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b Nov 01 22:31:07.002006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.001968 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" event={"ID":"eca5bc60-4778-4bca-be96-3ff971cde016","Type":"ContainerStarted","Data":"28416cd3eaf4e467776798bd9b03361ac8217cd0de60de72877332c3c362cfa5"} Nov 01 22:31:07.002456 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.002082 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" event={"ID":"eca5bc60-4778-4bca-be96-3ff971cde016","Type":"ContainerStarted","Data":"3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b"} Nov 01 22:31:07.024738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.024690 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" podStartSLOduration=2.024676306 podStartE2EDuration="2.024676306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:07.02072055 +0000 UTC m=+2813.994623089" watchObservedRunningTime="2026-11-01 22:31:07.024676306 +0000 UTC m=+2813.998578847" Nov 01 22:31:07.160391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.160363 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:07.221825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.221796 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\") pod \"02353b4a-6603-4af8-8f49-161b455410c9\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " Nov 01 22:31:07.222060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.222017 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\") pod \"02353b4a-6603-4af8-8f49-161b455410c9\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " Nov 01 22:31:07.222159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.222078 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\" (UniqueName: \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\") pod \"02353b4a-6603-4af8-8f49-161b455410c9\" (UID: \"02353b4a-6603-4af8-8f49-161b455410c9\") " Nov 01 22:31:07.222241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.222214 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca" pod "02353b4a-6603-4af8-8f49-161b455410c9" (UID: "02353b4a-6603-4af8-8f49-161b455410c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:07.222731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.222699 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:07.224620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.224590 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca" pod "02353b4a-6603-4af8-8f49-161b455410c9" (UID: "02353b4a-6603-4af8-8f49-161b455410c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:07.224802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.224783 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw" pod "02353b4a-6603-4af8-8f49-161b455410c9" (UID: "02353b4a-6603-4af8-8f49-161b455410c9"). InnerVolumeSpecName "kube-api-access-fhrmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:07.323911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.323886 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02353b4a-6603-4af8-8f49-161b455410c9-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:07.323911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:07.323912 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrmw\" (UniqueName: \"kubernetes.io/projected/02353b4a-6603-4af8-8f49-161b455410c9-kube-api-access-fhrmw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:08.011106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.011076 4395 generic.go:356] "Generic (PLEG): container finished" podID="eca5bc60-4778-4bca-be96-3ff971cde016" containerID="28416cd3eaf4e467776798bd9b03361ac8217cd0de60de72877332c3c362cfa5" exitCode=0 Nov 01 22:31:08.011585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.011163 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" event={"ID":"eca5bc60-4778-4bca-be96-3ff971cde016","Type":"ContainerDied","Data":"28416cd3eaf4e467776798bd9b03361ac8217cd0de60de72877332c3c362cfa5"} Nov 01 22:31:08.012646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.012631 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" Nov 01 22:31:08.012723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.012669 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw" event={"ID":"02353b4a-6603-4af8-8f49-161b455410c9","Type":"ContainerDied","Data":"ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08"} Nov 01 22:31:08.012723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.012711 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd43a4ba58a2c518790120388f3192b14d400bec2231911f48a0ded54b4ef08" Nov 01 22:31:08.072479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.069017 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:31:08.072479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.069983 4395 cpu_manager.go:414] "containerMap: removing container" podUID="02353b4a-6603-4af8-8f49-161b455410c9" containerName="revoke-keys" Nov 01 22:31:08.072479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.070001 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02353b4a-6603-4af8-8f49-161b455410c9" containerName="revoke-keys" Nov 01 22:31:08.072479 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.070253 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="02353b4a-6603-4af8-8f49-161b455410c9" containerName="revoke-keys" Nov 01 22:31:08.079610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.074700 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.096084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.083567 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-09656e-openshift-default-dockercfg-j7hg2\"" Nov 01 22:31:08.096084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.094184 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:31:08.134013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.133974 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134013 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134040 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134181 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134225 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134301 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69hd\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134362 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134399 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.134474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.135003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.134618 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235320 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235290 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235327 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235351 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235369 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235415 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69hd\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235446 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235475 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235605 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235747 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235778 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235797 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.235840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235817 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.236148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.235979 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.236379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.236361 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.237905 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.237887 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.238070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.238054 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.248120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.248097 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.248510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.248493 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69hd\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.410413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.410326 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:08.730938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.730915 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:31:08.731483 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:08.731457 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b36179e_0951_4274_baaa_b468ed45bffe.slice/crio-38b74e739d370ee1528260f33e4ceebd963ab27911b4804d846ac4cf684f2bb2 WatchSource:0}: Error finding container 38b74e739d370ee1528260f33e4ceebd963ab27911b4804d846ac4cf684f2bb2: Status 404 returned error can't find the container with id 38b74e739d370ee1528260f33e4ceebd963ab27911b4804d846ac4cf684f2bb2 Nov 01 22:31:08.734058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.734006 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:31:08.734162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.734105 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:31:08.734162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:08.734150 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:31:09.027197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.026818 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" event={"ID":"0b36179e-0951-4274-baaa-b468ed45bffe","Type":"ContainerStarted","Data":"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e"} Nov 01 22:31:09.027197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.026879 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" event={"ID":"0b36179e-0951-4274-baaa-b468ed45bffe","Type":"ContainerStarted","Data":"38b74e739d370ee1528260f33e4ceebd963ab27911b4804d846ac4cf684f2bb2"} Nov 01 22:31:09.057954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.057892 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podStartSLOduration=1.057878006 podStartE2EDuration="1.057878006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:09.05741266 +0000 UTC m=+2816.031315198" watchObservedRunningTime="2026-11-01 22:31:09.057878006 +0000 UTC m=+2816.031780545" Nov 01 22:31:09.196728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.196701 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:09.248660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.248631 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\") pod \"eca5bc60-4778-4bca-be96-3ff971cde016\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " Nov 01 22:31:09.248853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.248683 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\") pod \"eca5bc60-4778-4bca-be96-3ff971cde016\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " Nov 01 22:31:09.248853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.248723 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\" (UniqueName: \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\") pod \"eca5bc60-4778-4bca-be96-3ff971cde016\" (UID: \"eca5bc60-4778-4bca-be96-3ff971cde016\") " Nov 01 22:31:09.249147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.249118 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca" pod "eca5bc60-4778-4bca-be96-3ff971cde016" (UID: "eca5bc60-4778-4bca-be96-3ff971cde016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:09.249536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.249499 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.251222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.251191 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca" pod "eca5bc60-4778-4bca-be96-3ff971cde016" (UID: "eca5bc60-4778-4bca-be96-3ff971cde016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:09.251574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.251544 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb" pod "eca5bc60-4778-4bca-be96-3ff971cde016" (UID: "eca5bc60-4778-4bca-be96-3ff971cde016"). InnerVolumeSpecName "kube-api-access-jmgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:09.351135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.351104 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca5bc60-4778-4bca-be96-3ff971cde016-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.351135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.351130 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgjb\" (UniqueName: \"kubernetes.io/projected/eca5bc60-4778-4bca-be96-3ff971cde016-kube-api-access-jmgjb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:09.411537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.411505 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:09.412903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.412877 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:09.413062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:09.412924 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:10.049074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:10.044867 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" Nov 01 22:31:10.049074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:10.044934 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h" event={"ID":"eca5bc60-4778-4bca-be96-3ff971cde016","Type":"ContainerDied","Data":"3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b"} Nov 01 22:31:10.049074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:10.044979 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dbcae4e0787fc043dc306735829d98798a1e4f01580438f019062ec3a7daf3b" Nov 01 22:31:10.413306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:10.413225 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:10.413464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:10.413303 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:11.410986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:11.410950 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:11.411431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:11.411011 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:12.411102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:12.411059 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:12.411550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:12.411133 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:13.411107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:13.411065 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:13.411552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:13.411127 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:13.514707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:13.514674 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:31:13.514898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:13.514880 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" podUID="c15a7b03-3fb2-4009-be54-e820e315fcaf" containerName="payload-pre-processing" containerID="cri-o://4e340e5835faca17e483bf78385c48500c67c2db2f56e0fb30ea1ddc6af484d5" gracePeriod=30 Nov 01 22:31:14.009304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.009270 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j"] Nov 01 22:31:14.015694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.015640 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60807f-9c44j"] Nov 01 22:31:14.080844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.080540 4395 generic.go:356] "Generic (PLEG): container finished" podID="c15a7b03-3fb2-4009-be54-e820e315fcaf" containerID="4e340e5835faca17e483bf78385c48500c67c2db2f56e0fb30ea1ddc6af484d5" exitCode=0 Nov 01 22:31:14.080844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.080714 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" event={"ID":"c15a7b03-3fb2-4009-be54-e820e315fcaf","Type":"ContainerDied","Data":"4e340e5835faca17e483bf78385c48500c67c2db2f56e0fb30ea1ddc6af484d5"} Nov 01 22:31:14.162792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.162761 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:31:14.265729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.265686 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\") pod \"c15a7b03-3fb2-4009-be54-e820e315fcaf\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " Nov 01 22:31:14.265925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.265775 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\" (UniqueName: \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\") pod \"c15a7b03-3fb2-4009-be54-e820e315fcaf\" (UID: \"c15a7b03-3fb2-4009-be54-e820e315fcaf\") " Nov 01 22:31:14.267400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.267349 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume" pod "c15a7b03-3fb2-4009-be54-e820e315fcaf" (UID: "c15a7b03-3fb2-4009-be54-e820e315fcaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:14.271654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.271607 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97" pod "c15a7b03-3fb2-4009-be54-e820e315fcaf" (UID: "c15a7b03-3fb2-4009-be54-e820e315fcaf"). InnerVolumeSpecName "kube-api-access-mrj97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:14.369094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.367716 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15a7b03-3fb2-4009-be54-e820e315fcaf-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:14.369094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.367749 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrj97\" (UniqueName: \"kubernetes.io/projected/c15a7b03-3fb2-4009-be54-e820e315fcaf-kube-api-access-mrj97\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:14.462053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.461925 4395 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Nov 01 22:31:14.462053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.461990 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Nov 01 22:31:14.568563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.568529 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp"] Nov 01 22:31:14.569150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569135 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eca5bc60-4778-4bca-be96-3ff971cde016" containerName="revoke-keys" Nov 01 22:31:14.569150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569150 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca5bc60-4778-4bca-be96-3ff971cde016" containerName="revoke-keys" Nov 01 22:31:14.569281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569183 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c15a7b03-3fb2-4009-be54-e820e315fcaf" containerName="payload-pre-processing" Nov 01 22:31:14.569281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569189 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15a7b03-3fb2-4009-be54-e820e315fcaf" containerName="payload-pre-processing" Nov 01 22:31:14.569281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569273 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15a7b03-3fb2-4009-be54-e820e315fcaf" containerName="payload-pre-processing" Nov 01 22:31:14.569425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.569287 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca5bc60-4778-4bca-be96-3ff971cde016" containerName="revoke-keys" Nov 01 22:31:14.572791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.572772 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.606437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.604064 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp"] Nov 01 22:31:14.664200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.664110 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:14.664200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.664112 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:14.672753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.672718 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.672942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.672868 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq99q\" (UniqueName: \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.673075 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.673053 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.774452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.774396 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.774868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.774843 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.775123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.775103 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq99q\" (UniqueName: \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.776466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.776439 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.779874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.779823 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.813047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.810702 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq99q\" (UniqueName: \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:14.912054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:14.910537 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:15.090944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.090914 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" Nov 01 22:31:15.091184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.091052 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh" event={"ID":"c15a7b03-3fb2-4009-be54-e820e315fcaf","Type":"ContainerDied","Data":"d5f639521b8d888e2f403a08536bcb4c6412fe6e1222fd0d93f95c8b7588d454"} Nov 01 22:31:15.091184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.091107 4395 scope.go:122] "RemoveContainer" containerID="4e340e5835faca17e483bf78385c48500c67c2db2f56e0fb30ea1ddc6af484d5" Nov 01 22:31:15.125393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.125367 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:31:15.130736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.130702 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-cb55749f-rddvh"] Nov 01 22:31:15.164845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.164820 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba8ee61-175d-488f-8022-9f782d68a8fa" path="/var/lib/kubelet/pods/8ba8ee61-175d-488f-8022-9f782d68a8fa/volumes" Nov 01 22:31:15.165234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.165222 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15a7b03-3fb2-4009-be54-e820e315fcaf" path="/var/lib/kubelet/pods/c15a7b03-3fb2-4009-be54-e820e315fcaf/volumes" Nov 01 22:31:15.229385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.229358 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp"] Nov 01 22:31:15.230566 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:15.230541 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e0721a_fadf_4d6c_a030_f6d06ac5b80b.slice/crio-f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13 WatchSource:0}: Error finding container f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13: Status 404 returned error can't find the container with id f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13 Nov 01 22:31:15.661295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.661093 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:15.661733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.661624 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:15.663077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:15.663016 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:31:16.101525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:16.101485 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" event={"ID":"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b","Type":"ContainerStarted","Data":"9631efd6d024e4b4b5e52ee8edfef5ee02443b036999dc828ca63d24e7456a50"} Nov 01 22:31:16.101713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:16.101550 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" event={"ID":"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b","Type":"ContainerStarted","Data":"f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13"} Nov 01 22:31:16.120721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:16.120676 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" podStartSLOduration=2.120663128 podStartE2EDuration="2.120663128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:16.118864113 +0000 UTC m=+2823.092766657" watchObservedRunningTime="2026-11-01 22:31:16.120663128 +0000 UTC m=+2823.094565666" Nov 01 22:31:17.113529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.113493 4395 generic.go:356] "Generic (PLEG): container finished" podID="d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" containerID="9631efd6d024e4b4b5e52ee8edfef5ee02443b036999dc828ca63d24e7456a50" exitCode=0 Nov 01 22:31:17.113855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.113573 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" event={"ID":"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b","Type":"ContainerDied","Data":"9631efd6d024e4b4b5e52ee8edfef5ee02443b036999dc828ca63d24e7456a50"} Nov 01 22:31:17.588388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.588347 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:31:17.588629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.588607 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" podUID="36f2108a-798d-4342-a93b-b1af56582e90" containerName="maas-api" containerID="cri-o://9bb97c9916c671ec1405de43067362d8e27edcccdb1ec02f45031c49f32f430b" gracePeriod=30 Nov 01 22:31:17.678166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.678116 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:31:17.678556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:17.678404 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" podUID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" containerName="payload-pre-processing" containerID="cri-o://80410bbfa96e9f9668473e974381a707ce4874cdaa6ad3ae361e5f2926d80657" gracePeriod=30 Nov 01 22:31:18.184122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.181288 4395 generic.go:356] "Generic (PLEG): container finished" podID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" containerID="80410bbfa96e9f9668473e974381a707ce4874cdaa6ad3ae361e5f2926d80657" exitCode=0 Nov 01 22:31:18.184122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.181403 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" event={"ID":"2a4330ae-04f2-4189-9fc2-09c72940a5e2","Type":"ContainerDied","Data":"80410bbfa96e9f9668473e974381a707ce4874cdaa6ad3ae361e5f2926d80657"} Nov 01 22:31:18.184122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.181453 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" event={"ID":"2a4330ae-04f2-4189-9fc2-09c72940a5e2","Type":"ContainerDied","Data":"c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453"} Nov 01 22:31:18.184122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.181477 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c193f8697cb2857a10d05cbfb0a1261e626fca066e9c75f20fd59522496be453" Nov 01 22:31:18.189599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.189381 4395 generic.go:356] "Generic (PLEG): container finished" podID="36f2108a-798d-4342-a93b-b1af56582e90" containerID="9bb97c9916c671ec1405de43067362d8e27edcccdb1ec02f45031c49f32f430b" exitCode=0 Nov 01 22:31:18.189784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.189687 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" event={"ID":"36f2108a-798d-4342-a93b-b1af56582e90","Type":"ContainerDied","Data":"9bb97c9916c671ec1405de43067362d8e27edcccdb1ec02f45031c49f32f430b"} Nov 01 22:31:18.203656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.203623 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:31:18.292020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.290822 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\" (UniqueName: \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\") pod \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " Nov 01 22:31:18.292020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.290970 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\") pod \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\" (UID: \"2a4330ae-04f2-4189-9fc2-09c72940a5e2\") " Nov 01 22:31:18.292294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.292079 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume" pod "2a4330ae-04f2-4189-9fc2-09c72940a5e2" (UID: "2a4330ae-04f2-4189-9fc2-09c72940a5e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:18.298909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.298573 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9" pod "2a4330ae-04f2-4189-9fc2-09c72940a5e2" (UID: "2a4330ae-04f2-4189-9fc2-09c72940a5e2"). InnerVolumeSpecName "kube-api-access-7thf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:18.333313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.332986 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:31:18.415041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.414927 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thf9\" (UniqueName: \"kubernetes.io/projected/2a4330ae-04f2-4189-9fc2-09c72940a5e2-kube-api-access-7thf9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.415041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.414962 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4330ae-04f2-4189-9fc2-09c72940a5e2-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.433680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.433650 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:18.516009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.515973 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\") pod \"36f2108a-798d-4342-a93b-b1af56582e90\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " Nov 01 22:31:18.516253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.516232 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\" (UniqueName: \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\") pod \"36f2108a-798d-4342-a93b-b1af56582e90\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " Nov 01 22:31:18.516369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.516303 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\") pod \"36f2108a-798d-4342-a93b-b1af56582e90\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " Nov 01 22:31:18.516447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.516409 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\") pod \"36f2108a-798d-4342-a93b-b1af56582e90\" (UID: \"36f2108a-798d-4342-a93b-b1af56582e90\") " Nov 01 22:31:18.517432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.516783 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca" pod "36f2108a-798d-4342-a93b-b1af56582e90" (UID: "36f2108a-798d-4342-a93b-b1af56582e90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:18.517432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.517004 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f2108a-798d-4342-a93b-b1af56582e90-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.519048 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.519003 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs" pod "36f2108a-798d-4342-a93b-b1af56582e90" (UID: "36f2108a-798d-4342-a93b-b1af56582e90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:18.519383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.519350 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd" pod "36f2108a-798d-4342-a93b-b1af56582e90" (UID: "36f2108a-798d-4342-a93b-b1af56582e90"). InnerVolumeSpecName "kube-api-access-8gnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:18.519620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.519590 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls" pod "36f2108a-798d-4342-a93b-b1af56582e90" (UID: "36f2108a-798d-4342-a93b-b1af56582e90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:18.617865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.617836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\") pod \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " Nov 01 22:31:18.618076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.617896 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\" (UniqueName: \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\") pod \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " Nov 01 22:31:18.618155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618129 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\") pod \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\" (UID: \"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b\") " Nov 01 22:31:18.618232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618194 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca" pod "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" (UID: "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:18.618512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618493 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnmd\" (UniqueName: \"kubernetes.io/projected/36f2108a-798d-4342-a93b-b1af56582e90-kube-api-access-8gnmd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.618512 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618513 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.618669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618522 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f2108a-798d-4342-a93b-b1af56582e90-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.618669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.618533 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.620436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.620409 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q" pod "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" (UID: "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b"). InnerVolumeSpecName "kube-api-access-tq99q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:18.620436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.620414 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca" pod "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" (UID: "d7e0721a-fadf-4d6c-a030-f6d06ac5b80b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:18.719467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.719442 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq99q\" (UniqueName: \"kubernetes.io/projected/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-kube-api-access-tq99q\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:18.719575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:18.719469 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:19.197652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.197628 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" Nov 01 22:31:19.198105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.197663 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs" event={"ID":"36f2108a-798d-4342-a93b-b1af56582e90","Type":"ContainerDied","Data":"85bd54d3327616bc277ba5b4c1f0e345db83887fbc96b474a750b410ef742ba3"} Nov 01 22:31:19.198105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.197716 4395 scope.go:122] "RemoveContainer" containerID="9bb97c9916c671ec1405de43067362d8e27edcccdb1ec02f45031c49f32f430b" Nov 01 22:31:19.201504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.199799 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" Nov 01 22:31:19.201504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.200003 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp" event={"ID":"d7e0721a-fadf-4d6c-a030-f6d06ac5b80b","Type":"ContainerDied","Data":"f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13"} Nov 01 22:31:19.201504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.200095 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04bfc6d0a57c12bd3571594deeb3de591dd613dc39564bf73a78e3a59752e13" Nov 01 22:31:19.201504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.200190 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8" Nov 01 22:31:19.222319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.222296 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:31:19.227209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.227183 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-09656e-5557ffb5bc-p8bgs"] Nov 01 22:31:19.234255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.234236 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:31:19.238670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:19.238649 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-09656e-88b99879-zmdd8"] Nov 01 22:31:20.531770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:20.531738 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 22:31:20.532222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:20.532065 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" podUID="974aea98-ab66-46d1-803b-ef25ee14b91b" containerName="maas-api" containerID="cri-o://948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326" gracePeriod=30 Nov 01 22:31:20.874228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:20.873517 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh"] Nov 01 22:31:20.876007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:20.875974 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b0b7a3-wk8gh"] Nov 01 22:31:20.991771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:20.991745 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 22:31:21.065109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.065074 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\" (UniqueName: \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\") pod \"974aea98-ab66-46d1-803b-ef25ee14b91b\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " Nov 01 22:31:21.065280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.065177 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\") pod \"974aea98-ab66-46d1-803b-ef25ee14b91b\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " Nov 01 22:31:21.065280 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.065267 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\") pod \"974aea98-ab66-46d1-803b-ef25ee14b91b\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " Nov 01 22:31:21.065439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.065393 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\") pod \"974aea98-ab66-46d1-803b-ef25ee14b91b\" (UID: \"974aea98-ab66-46d1-803b-ef25ee14b91b\") " Nov 01 22:31:21.066676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.066619 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca" pod "974aea98-ab66-46d1-803b-ef25ee14b91b" (UID: "974aea98-ab66-46d1-803b-ef25ee14b91b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:21.068658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.068630 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc" pod "974aea98-ab66-46d1-803b-ef25ee14b91b" (UID: "974aea98-ab66-46d1-803b-ef25ee14b91b"). InnerVolumeSpecName "kube-api-access-8hwdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:21.069490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.069465 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls" pod "974aea98-ab66-46d1-803b-ef25ee14b91b" (UID: "974aea98-ab66-46d1-803b-ef25ee14b91b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:21.070406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.070369 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs" pod "974aea98-ab66-46d1-803b-ef25ee14b91b" (UID: "974aea98-ab66-46d1-803b-ef25ee14b91b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:21.166428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.166365 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" path="/var/lib/kubelet/pods/2a4330ae-04f2-4189-9fc2-09c72940a5e2/volumes" Nov 01 22:31:21.166984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.166965 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f2108a-798d-4342-a93b-b1af56582e90" path="/var/lib/kubelet/pods/36f2108a-798d-4342-a93b-b1af56582e90/volumes" Nov 01 22:31:21.167134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.167111 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:21.167214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.167140 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwdc\" (UniqueName: \"kubernetes.io/projected/974aea98-ab66-46d1-803b-ef25ee14b91b-kube-api-access-8hwdc\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:21.167214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.167157 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/974aea98-ab66-46d1-803b-ef25ee14b91b-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:21.167214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.167182 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/974aea98-ab66-46d1-803b-ef25ee14b91b-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:21.167468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.167454 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac09e354-e878-47b4-b771-98d0cf316203" path="/var/lib/kubelet/pods/ac09e354-e878-47b4-b771-98d0cf316203/volumes" Nov 01 22:31:21.229553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.229511 4395 generic.go:356] "Generic (PLEG): container finished" podID="974aea98-ab66-46d1-803b-ef25ee14b91b" containerID="948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326" exitCode=0 Nov 01 22:31:21.229769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.229559 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" Nov 01 22:31:21.229769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.229589 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" event={"ID":"974aea98-ab66-46d1-803b-ef25ee14b91b","Type":"ContainerDied","Data":"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326"} Nov 01 22:31:21.229769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.229646 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v" event={"ID":"974aea98-ab66-46d1-803b-ef25ee14b91b","Type":"ContainerDied","Data":"d6d182896dc7c943f6a0208cce4c4342cf2f6023d69d7447638d6b7a734259bd"} Nov 01 22:31:21.229769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.229667 4395 scope.go:122] "RemoveContainer" containerID="948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326" Nov 01 22:31:21.241543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.241520 4395 scope.go:122] "RemoveContainer" containerID="948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326" Nov 01 22:31:21.241888 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:21.241863 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326\": container with ID starting with 948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326 not found: ID does not exist" containerID="948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326" Nov 01 22:31:21.241986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.241898 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326"} err="failed to get container status \"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326\": rpc error: code = NotFound desc = could not find container \"948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326\": container with ID starting with 948e367eb13097456d7e3470a2a557ef5c08d19680e6130955f6f8a0f8a0a326 not found: ID does not exist" Nov 01 22:31:21.249168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.249140 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 22:31:21.253875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.253842 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-b0b7a3-66478c6f85-k585v"] Nov 01 22:31:21.411982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.411946 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:31:21.412432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.412217 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" containerID="cri-o://f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737" gracePeriod=30 Nov 01 22:31:21.556052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.552194 4395 patch_prober.go:29] interesting pod/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:31:21.556052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.552259 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:31:21.862978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.862896 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr"] Nov 01 22:31:21.863509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863495 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" containerName="revoke-keys" Nov 01 22:31:21.863563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863511 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" containerName="revoke-keys" Nov 01 22:31:21.863563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863532 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" containerName="payload-pre-processing" Nov 01 22:31:21.863563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863538 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" containerName="payload-pre-processing" Nov 01 22:31:21.863563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863549 4395 cpu_manager.go:414] "containerMap: removing container" podUID="974aea98-ab66-46d1-803b-ef25ee14b91b" containerName="maas-api" Nov 01 22:31:21.863563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863553 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974aea98-ab66-46d1-803b-ef25ee14b91b" containerName="maas-api" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863566 4395 cpu_manager.go:414] "containerMap: removing container" podUID="36f2108a-798d-4342-a93b-b1af56582e90" containerName="maas-api" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863573 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f2108a-798d-4342-a93b-b1af56582e90" containerName="maas-api" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863663 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4330ae-04f2-4189-9fc2-09c72940a5e2" containerName="payload-pre-processing" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863673 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="974aea98-ab66-46d1-803b-ef25ee14b91b" containerName="maas-api" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863680 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f2108a-798d-4342-a93b-b1af56582e90" containerName="maas-api" Nov 01 22:31:21.863728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.863689 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" containerName="revoke-keys" Nov 01 22:31:21.868605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.868587 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.877104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.875846 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5vv\" (UniqueName: \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.877104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.876208 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.877104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.876250 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.878312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.878289 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr"] Nov 01 22:31:21.882806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.882776 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:21.883018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.882981 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:21.976999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.976968 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.976999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.977003 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.977212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.977102 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5vv\" (UniqueName: \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.977769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.977744 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.979560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.979535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:21.989060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:21.989005 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5vv\" (UniqueName: \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:22.186843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:22.186764 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:22.310358 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:22.310315 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:31:22.467607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:22.467571 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr"] Nov 01 22:31:22.469141 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:22.469118 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda331eed0_5b43_47b8_aff6_dd9a94944f1b.slice/crio-1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977 WatchSource:0}: Error finding container 1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977: Status 404 returned error can't find the container with id 1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977 Nov 01 22:31:23.166095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:23.166004 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974aea98-ab66-46d1-803b-ef25ee14b91b" path="/var/lib/kubelet/pods/974aea98-ab66-46d1-803b-ef25ee14b91b/volumes" Nov 01 22:31:23.254800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:23.254761 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" event={"ID":"a331eed0-5b43-47b8-aff6-dd9a94944f1b","Type":"ContainerStarted","Data":"949a5753607ca236f0944c09354653a16038d43ace0cccc194296de0fdfa64bc"} Nov 01 22:31:23.254993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:23.254821 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" event={"ID":"a331eed0-5b43-47b8-aff6-dd9a94944f1b","Type":"ContainerStarted","Data":"1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977"} Nov 01 22:31:23.274496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:23.274443 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" podStartSLOduration=2.274427782 podStartE2EDuration="2.274427782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:23.271693871 +0000 UTC m=+2830.245596409" watchObservedRunningTime="2026-11-01 22:31:23.274427782 +0000 UTC m=+2830.248330351" Nov 01 22:31:24.263197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:24.263160 4395 generic.go:356] "Generic (PLEG): container finished" podID="a331eed0-5b43-47b8-aff6-dd9a94944f1b" containerID="949a5753607ca236f0944c09354653a16038d43ace0cccc194296de0fdfa64bc" exitCode=0 Nov 01 22:31:24.263636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:24.263256 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" event={"ID":"a331eed0-5b43-47b8-aff6-dd9a94944f1b","Type":"ContainerDied","Data":"949a5753607ca236f0944c09354653a16038d43ace0cccc194296de0fdfa64bc"} Nov 01 22:31:24.705803 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:24.705697 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:31:24.722019 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:24.721988 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:31:25.469741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.469713 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:25.530070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.529967 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\") pod \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " Nov 01 22:31:25.530070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.530060 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\" (UniqueName: \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\") pod \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " Nov 01 22:31:25.530294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.530181 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\") pod \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\" (UID: \"a331eed0-5b43-47b8-aff6-dd9a94944f1b\") " Nov 01 22:31:25.530475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.530444 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca" pod "a331eed0-5b43-47b8-aff6-dd9a94944f1b" (UID: "a331eed0-5b43-47b8-aff6-dd9a94944f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:25.530828 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.530799 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.532527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.532504 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca" pod "a331eed0-5b43-47b8-aff6-dd9a94944f1b" (UID: "a331eed0-5b43-47b8-aff6-dd9a94944f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:25.532613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.532567 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv" pod "a331eed0-5b43-47b8-aff6-dd9a94944f1b" (UID: "a331eed0-5b43-47b8-aff6-dd9a94944f1b"). InnerVolumeSpecName "kube-api-access-tt5vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:25.631851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.631825 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5vv\" (UniqueName: \"kubernetes.io/projected/a331eed0-5b43-47b8-aff6-dd9a94944f1b-kube-api-access-tt5vv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:25.631851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:25.631852 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a331eed0-5b43-47b8-aff6-dd9a94944f1b-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.305152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.305115 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" event={"ID":"a331eed0-5b43-47b8-aff6-dd9a94944f1b","Type":"ContainerDied","Data":"1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977"} Nov 01 22:31:26.305436 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.305421 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dfa59eb27776cdd3b5ec4653397166cb99b5a6ebc884ee0ef27bb2c270e8977" Nov 01 22:31:26.305715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.305696 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr" Nov 01 22:31:26.314165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.314132 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp"] Nov 01 22:31:26.320782 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.320749 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-521734-zd8pp"] Nov 01 22:31:26.720886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.720863 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:31:26.760283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760251 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760359 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760413 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760522 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:26.760637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760583 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760649 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760684 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760688 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:26.760942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760785 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760854 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.760942 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760902 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:26.761145 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.760946 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\") pod \"cefce3b9-edf9-402c-a781-2ad03e534677\" (UID: \"cefce3b9-edf9-402c-a781-2ad03e534677\") " Nov 01 22:31:26.761489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761146 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:26.761767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761743 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:26.762071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761836 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.762071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761858 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.762071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761874 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.762071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.761890 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.763801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.763774 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:26.764768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.764744 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:26.765384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.765358 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:26.766044 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.766001 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr" pod "cefce3b9-edf9-402c-a781-2ad03e534677" (UID: "cefce3b9-edf9-402c-a781-2ad03e534677"). InnerVolumeSpecName "kube-api-access-jmctr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:26.862842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.862809 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefce3b9-edf9-402c-a781-2ad03e534677-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.862842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.862837 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefce3b9-edf9-402c-a781-2ad03e534677-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.862842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.862847 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmctr\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-kube-api-access-jmctr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.863123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.862856 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefce3b9-edf9-402c-a781-2ad03e534677-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:26.863123 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:26.862865 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefce3b9-edf9-402c-a781-2ad03e534677-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:27.165923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.165888 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e0721a-fadf-4d6c-a030-f6d06ac5b80b" path="/var/lib/kubelet/pods/d7e0721a-fadf-4d6c-a030-f6d06ac5b80b/volumes" Nov 01 22:31:27.313101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.313061 4395 generic.go:356] "Generic (PLEG): container finished" podID="cefce3b9-edf9-402c-a781-2ad03e534677" containerID="f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737" exitCode=0 Nov 01 22:31:27.313281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.313124 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" Nov 01 22:31:27.313281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.313136 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerDied","Data":"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737"} Nov 01 22:31:27.313281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.313194 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd" event={"ID":"cefce3b9-edf9-402c-a781-2ad03e534677","Type":"ContainerDied","Data":"f7caa3f9344a88b83f6fae0cc973b7ded6176a1d037cccc107f8f1bdc9a13f0b"} Nov 01 22:31:27.313281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.313224 4395 scope.go:122] "RemoveContainer" containerID="f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737" Nov 01 22:31:27.324471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.324455 4395 scope.go:122] "RemoveContainer" containerID="563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372" Nov 01 22:31:27.333265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.333241 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:31:27.334176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.334162 4395 scope.go:122] "RemoveContainer" containerID="f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737" Nov 01 22:31:27.334478 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:27.334456 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737\": container with ID starting with f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737 not found: ID does not exist" containerID="f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737" Nov 01 22:31:27.334555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.334490 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737"} err="failed to get container status \"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737\": rpc error: code = NotFound desc = could not find container \"f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737\": container with ID starting with f0aa3d424dc298f5e8a62ef9140018e06931e6de67de1c86574b7ec1b838a737 not found: ID does not exist" Nov 01 22:31:27.334555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.334514 4395 scope.go:122] "RemoveContainer" containerID="563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372" Nov 01 22:31:27.334815 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:27.334785 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372\": container with ID starting with 563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372 not found: ID does not exist" containerID="563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372" Nov 01 22:31:27.334862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.334822 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372"} err="failed to get container status \"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372\": rpc error: code = NotFound desc = could not find container \"563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372\": container with ID starting with 563f7397329ecda1409b53dd5517f6f4f318bb3d80602facab45a69e7e192372 not found: ID does not exist" Nov 01 22:31:27.339705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:27.339686 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-60807f-openshift-default-7b76b969c9-xnljd"] Nov 01 22:31:28.314088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:28.303570 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 22:31:28.314088 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:28.303772 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" containerID="cri-o://05cc74160f957677e50da61e64cebd842103217efe59e401dec975b0508dd44a" gracePeriod=30 Nov 01 22:31:29.165982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:29.165939 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" path="/var/lib/kubelet/pods/cefce3b9-edf9-402c-a781-2ad03e534677/volumes" Nov 01 22:31:29.706500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:29.706446 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" containerID="cri-o://c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899" gracePeriod=25 Nov 01 22:31:29.722897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:29.722851 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" containerID="cri-o://0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1" gracePeriod=25 Nov 01 22:31:30.911661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:30.911615 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:30.911661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:30.911653 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:32.023268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:32.023226 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:31:32.537089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:32.537058 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr"] Nov 01 22:31:32.541376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:32.541348 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1783e3-769hr"] Nov 01 22:31:33.164752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.164721 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a331eed0-5b43-47b8-aff6-dd9a94944f1b" path="/var/lib/kubelet/pods/a331eed0-5b43-47b8-aff6-dd9a94944f1b/volumes" Nov 01 22:31:33.510058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.508286 4395 generic.go:356] "Generic (PLEG): container finished" podID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerID="05cc74160f957677e50da61e64cebd842103217efe59e401dec975b0508dd44a" exitCode=0 Nov 01 22:31:33.510058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.508468 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" event={"ID":"c3963ac0-ee4e-49ba-9dc0-4c76692b8176","Type":"ContainerDied","Data":"05cc74160f957677e50da61e64cebd842103217efe59e401dec975b0508dd44a"} Nov 01 22:31:33.670866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.670135 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:31:33.670866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.670366 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" containerID="cri-o://37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108" gracePeriod=30 Nov 01 22:31:33.675180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.675085 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:31:33.676574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.676506 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" containerID="cri-o://bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3" gracePeriod=30 Nov 01 22:31:33.679345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.679309 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:31:33.679530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.679508 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" containerID="cri-o://7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d" gracePeriod=30 Nov 01 22:31:33.705228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.705208 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790443 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790507 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790643 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790684 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790721 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790777 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790822 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.790906 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.793255 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\") pod \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\" (UID: \"c3963ac0-ee4e-49ba-9dc0-4c76692b8176\") " Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.794307 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:33.796352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.794481 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:33.809315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.798107 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:33.809315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.802226 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:33.818050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.814115 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:33.840049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.822501 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:33.840049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.830713 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "kube-api-access-7svms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:33.840049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.830941 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:33.840049 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.831170 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo" pod "c3963ac0-ee4e-49ba-9dc0-4c76692b8176" (UID: "c3963ac0-ee4e-49ba-9dc0-4c76692b8176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:33.895246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895213 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895240 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svms\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-kube-api-access-7svms\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895251 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895259 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895267 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895275 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895283 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895292 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:33.895525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:33.895299 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3963ac0-ee4e-49ba-9dc0-4c76692b8176-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.185990 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb"] Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186861 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186881 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186930 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186939 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186986 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a331eed0-5b43-47b8-aff6-dd9a94944f1b" containerName="revoke-keys" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.186994 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a331eed0-5b43-47b8-aff6-dd9a94944f1b" containerName="revoke-keys" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.187163 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a331eed0-5b43-47b8-aff6-dd9a94944f1b" containerName="revoke-keys" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.187178 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.187187 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:34.190598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.187203 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" containerName="istio-proxy" Nov 01 22:31:34.193679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.193656 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.205853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.205823 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb"] Nov 01 22:31:34.210622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.210590 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:34.217007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.211246 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:34.307588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.307552 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.307770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.307625 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.307770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.307715 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xll\" (UniqueName: \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.408716 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.408685 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.408880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.408725 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xll\" (UniqueName: \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.408880 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.408852 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.409525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.409494 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.411554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.411527 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.421420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.421395 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xll\" (UniqueName: \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.515995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.515965 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" Nov 01 22:31:34.516188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.515990 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn" event={"ID":"c3963ac0-ee4e-49ba-9dc0-4c76692b8176","Type":"ContainerDied","Data":"d1cbf786a609fbfee11c22a028a0b965cabded6da1233e20c8aac36bba53e108"} Nov 01 22:31:34.516188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.516076 4395 scope.go:122] "RemoveContainer" containerID="05cc74160f957677e50da61e64cebd842103217efe59e401dec975b0508dd44a" Nov 01 22:31:34.524297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.524274 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:34.544368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.544343 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 22:31:34.551712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.551689 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-bnccn"] Nov 01 22:31:34.808335 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:34.808305 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb"] Nov 01 22:31:34.809437 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:34.809412 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da28f65_de84_410e_a58d_245c62b73a2a.slice/crio-35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40 WatchSource:0}: Error finding container 35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40: Status 404 returned error can't find the container with id 35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40 Nov 01 22:31:35.164006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:35.163981 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3963ac0-ee4e-49ba-9dc0-4c76692b8176" path="/var/lib/kubelet/pods/c3963ac0-ee4e-49ba-9dc0-4c76692b8176/volumes" Nov 01 22:31:35.523347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:35.523305 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" event={"ID":"3da28f65-de84-410e-a58d-245c62b73a2a","Type":"ContainerStarted","Data":"a592790c9f3ca44976674485c9fcbaa9e1b60d4fbd97cb1d768c706a5901197c"} Nov 01 22:31:35.523809 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:35.523365 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" event={"ID":"3da28f65-de84-410e-a58d-245c62b73a2a","Type":"ContainerStarted","Data":"35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40"} Nov 01 22:31:35.543276 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:35.543231 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" podStartSLOduration=1.5432179129999999 podStartE2EDuration="1.543217913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:35.54080249 +0000 UTC m=+2842.514705029" watchObservedRunningTime="2026-11-01 22:31:35.543217913 +0000 UTC m=+2842.517120500" Nov 01 22:31:36.533808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:36.533767 4395 generic.go:356] "Generic (PLEG): container finished" podID="3da28f65-de84-410e-a58d-245c62b73a2a" containerID="a592790c9f3ca44976674485c9fcbaa9e1b60d4fbd97cb1d768c706a5901197c" exitCode=0 Nov 01 22:31:36.534224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:36.533845 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" event={"ID":"3da28f65-de84-410e-a58d-245c62b73a2a","Type":"ContainerDied","Data":"a592790c9f3ca44976674485c9fcbaa9e1b60d4fbd97cb1d768c706a5901197c"} Nov 01 22:31:37.025201 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.025150 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" containerID="cri-o://1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658" gracePeriod=25 Nov 01 22:31:37.696153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.696126 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:37.866311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.865888 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\" (UniqueName: \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\") pod \"3da28f65-de84-410e-a58d-245c62b73a2a\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " Nov 01 22:31:37.866311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.865996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\") pod \"3da28f65-de84-410e-a58d-245c62b73a2a\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " Nov 01 22:31:37.866311 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.866261 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\") pod \"3da28f65-de84-410e-a58d-245c62b73a2a\" (UID: \"3da28f65-de84-410e-a58d-245c62b73a2a\") " Nov 01 22:31:37.867004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.866974 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca" pod "3da28f65-de84-410e-a58d-245c62b73a2a" (UID: "3da28f65-de84-410e-a58d-245c62b73a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:37.869415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.869385 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll" pod "3da28f65-de84-410e-a58d-245c62b73a2a" (UID: "3da28f65-de84-410e-a58d-245c62b73a2a"). InnerVolumeSpecName "kube-api-access-b5xll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:37.870689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.870412 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca" pod "3da28f65-de84-410e-a58d-245c62b73a2a" (UID: "3da28f65-de84-410e-a58d-245c62b73a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:37.968511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.968478 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:37.968511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.968506 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xll\" (UniqueName: \"kubernetes.io/projected/3da28f65-de84-410e-a58d-245c62b73a2a-kube-api-access-b5xll\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:37.968511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:37.968517 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da28f65-de84-410e-a58d-245c62b73a2a-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:38.548096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.548069 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" Nov 01 22:31:38.548269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.548066 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb" event={"ID":"3da28f65-de84-410e-a58d-245c62b73a2a","Type":"ContainerDied","Data":"35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40"} Nov 01 22:31:38.548269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.548215 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35051960099d108bfff0de458ae75dec2903f86053b8acfa1ca4a78ec002be40" Nov 01 22:31:38.968611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.968580 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:31:38.987669 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987644 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987687 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987810 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987866 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987897 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.987939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987920 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.988473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.987978 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:38.990761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.990664 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.990761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.990724 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.991281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.991256 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.991577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.991556 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:38.991967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.991648 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:38.998416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:38.997442 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.009742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.009705 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.010069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.010022 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "kube-api-access-864fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.017844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.017548 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:31:39.044749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.041758 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:31:39.089283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089248 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.089463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089306 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.089463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089367 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089397 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089429 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089459 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\") pod \"60a14cb7-9703-416a-9730-83950bb34b08\" (UID: \"60a14cb7-9703-416a-9730-83950bb34b08\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089490 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089533 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089576 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089617 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089644 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089710 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.089817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089770 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089822 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089855 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089899 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089931 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089959 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.089998 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090014 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090178 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\") pod \"31968d07-cb32-46ad-a888-d979e624c5e0\" (UID: \"31968d07-cb32-46ad-a888-d979e624c5e0\") " Nov 01 22:31:39.090238 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090233 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\") pod \"de6e0519-0259-4f2b-bb4e-9179ae59f022\" (UID: \"de6e0519-0259-4f2b-bb4e-9179ae59f022\") " Nov 01 22:31:39.090683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090301 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090362 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090507 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090564 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090774 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090864 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090886 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090900 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a14cb7-9703-416a-9730-83950bb34b08-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090914 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a14cb7-9703-416a-9730-83950bb34b08-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090926 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090940 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.090961 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090954 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090968 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090981 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6e0519-0259-4f2b-bb4e-9179ae59f022-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090994 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091006 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091018 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091048 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091061 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864fk\" (UniqueName: \"kubernetes.io/projected/60a14cb7-9703-416a-9730-83950bb34b08-kube-api-access-864fk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091074 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.090950 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data" pod "60a14cb7-9703-416a-9730-83950bb34b08" (UID: "60a14cb7-9703-416a-9730-83950bb34b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.091415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091332 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:39.091974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.091591 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.092785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.092757 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.094159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.094124 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "kube-api-access-hndpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.094455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.094427 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.095737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.095688 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:39.095737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.095703 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token" pod "de6e0519-0259-4f2b-bb4e-9179ae59f022" (UID: "de6e0519-0259-4f2b-bb4e-9179ae59f022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.095737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.095737 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.096305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.096268 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:39.097475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.097450 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg" pod "31968d07-cb32-46ad-a888-d979e624c5e0" (UID: "31968d07-cb32-46ad-a888-d979e624c5e0"). InnerVolumeSpecName "kube-api-access-n2vvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:39.191932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191903 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.191932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191933 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191950 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191962 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a14cb7-9703-416a-9730-83950bb34b08-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191975 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndpk\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-kube-api-access-hndpk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.191987 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31968d07-cb32-46ad-a888-d979e624c5e0-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192000 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31968d07-cb32-46ad-a888-d979e624c5e0-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192012 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6e0519-0259-4f2b-bb4e-9179ae59f022-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192045 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vvg\" (UniqueName: \"kubernetes.io/projected/31968d07-cb32-46ad-a888-d979e624c5e0-kube-api-access-n2vvg\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192059 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192072 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31968d07-cb32-46ad-a888-d979e624c5e0-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.192171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.192084 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6e0519-0259-4f2b-bb4e-9179ae59f022-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:39.555726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.555681 4395 generic.go:356] "Generic (PLEG): container finished" podID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerID="7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d" exitCode=0 Nov 01 22:31:39.555914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.555729 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" Nov 01 22:31:39.555914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.555763 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" event={"ID":"de6e0519-0259-4f2b-bb4e-9179ae59f022","Type":"ContainerDied","Data":"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d"} Nov 01 22:31:39.555914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.555830 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5" event={"ID":"de6e0519-0259-4f2b-bb4e-9179ae59f022","Type":"ContainerDied","Data":"42cbbde1e79a385c1066320c93571eac482db45b956b123f85f1eeba5ba44bba"} Nov 01 22:31:39.555914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.555859 4395 scope.go:122] "RemoveContainer" containerID="7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d" Nov 01 22:31:39.557345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.557321 4395 generic.go:356] "Generic (PLEG): container finished" podID="60a14cb7-9703-416a-9730-83950bb34b08" containerID="37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108" exitCode=0 Nov 01 22:31:39.557439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.557429 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" Nov 01 22:31:39.557500 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.557461 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" event={"ID":"60a14cb7-9703-416a-9730-83950bb34b08","Type":"ContainerDied","Data":"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108"} Nov 01 22:31:39.557552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.557508 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k" event={"ID":"60a14cb7-9703-416a-9730-83950bb34b08","Type":"ContainerDied","Data":"48a5f9053c51eba6d3f82b1b0b6f2bb63f033eec99af4946befab8553fa37966"} Nov 01 22:31:39.559273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.559248 4395 generic.go:356] "Generic (PLEG): container finished" podID="31968d07-cb32-46ad-a888-d979e624c5e0" containerID="bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3" exitCode=0 Nov 01 22:31:39.559378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.559296 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" event={"ID":"31968d07-cb32-46ad-a888-d979e624c5e0","Type":"ContainerDied","Data":"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3"} Nov 01 22:31:39.559378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.559355 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" event={"ID":"31968d07-cb32-46ad-a888-d979e624c5e0","Type":"ContainerDied","Data":"b52aa9747dbbc63b7a9f6f36b7e399f4bf759b637c24d2dffb08157b54a1108f"} Nov 01 22:31:39.559584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.559540 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2" Nov 01 22:31:39.569302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.569282 4395 scope.go:122] "RemoveContainer" containerID="7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d" Nov 01 22:31:39.569836 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:39.569814 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d\": container with ID starting with 7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d not found: ID does not exist" containerID="7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d" Nov 01 22:31:39.569971 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.569951 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d"} err="failed to get container status \"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d\": rpc error: code = NotFound desc = could not find container \"7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d\": container with ID starting with 7a2ee445023b829f8d6540cf4f980470623fa59e2560474dfe5e0b2783f0d81d not found: ID does not exist" Nov 01 22:31:39.570097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.570084 4395 scope.go:122] "RemoveContainer" containerID="37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108" Nov 01 22:31:39.581121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.581100 4395 scope.go:122] "RemoveContainer" containerID="37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108" Nov 01 22:31:39.581472 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:39.581438 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108\": container with ID starting with 37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108 not found: ID does not exist" containerID="37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108" Nov 01 22:31:39.581573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.581471 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108"} err="failed to get container status \"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108\": rpc error: code = NotFound desc = could not find container \"37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108\": container with ID starting with 37ef2e6e094752f07a3facebd4804282c5a6c7d2319853c6fd9d88a236093108 not found: ID does not exist" Nov 01 22:31:39.581573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.581488 4395 scope.go:122] "RemoveContainer" containerID="bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3" Nov 01 22:31:39.583301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.583278 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:31:39.587768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.587746 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-9twt5"] Nov 01 22:31:39.592694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.592669 4395 scope.go:122] "RemoveContainer" containerID="bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3" Nov 01 22:31:39.593122 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:39.593021 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3\": container with ID starting with bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3 not found: ID does not exist" containerID="bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3" Nov 01 22:31:39.593122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.593070 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3"} err="failed to get container status \"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3\": rpc error: code = NotFound desc = could not find container \"bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3\": container with ID starting with bc7d885880d91c0ad05c7133d7b91cda257dc915e414b6c49e3f4c555f9898a3 not found: ID does not exist" Nov 01 22:31:39.599385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.599362 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:31:39.605810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.605786 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-h428k"] Nov 01 22:31:39.616966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.616944 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:31:39.622095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:39.622073 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-gq9s2"] Nov 01 22:31:40.911359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:40.911265 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:40.911781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:40.911290 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:41.008411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.008380 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:31:41.008662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.008640 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" containerID="cri-o://1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151" gracePeriod=30 Nov 01 22:31:41.013799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.013768 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:31:41.014020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.013992 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" containerID="cri-o://54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0" gracePeriod=30 Nov 01 22:31:41.025543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.025510 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:31:41.025766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.025742 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" containerID="cri-o://c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920" gracePeriod=30 Nov 01 22:31:41.028623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.028598 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:31:41.028802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.028780 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" containerID="cri-o://00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c" gracePeriod=30 Nov 01 22:31:41.166213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.166128 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" path="/var/lib/kubelet/pods/31968d07-cb32-46ad-a888-d979e624c5e0/volumes" Nov 01 22:31:41.166657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.166642 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a14cb7-9703-416a-9730-83950bb34b08" path="/var/lib/kubelet/pods/60a14cb7-9703-416a-9730-83950bb34b08/volumes" Nov 01 22:31:41.167116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.167102 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" path="/var/lib/kubelet/pods/de6e0519-0259-4f2b-bb4e-9179ae59f022/volumes" Nov 01 22:31:41.453251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453176 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7"] Nov 01 22:31:41.453793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453769 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3da28f65-de84-410e-a58d-245c62b73a2a" containerName="revoke-keys" Nov 01 22:31:41.453793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453789 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da28f65-de84-410e-a58d-245c62b73a2a" containerName="revoke-keys" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453807 4395 cpu_manager.go:414] "containerMap: removing container" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453813 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453846 4395 cpu_manager.go:414] "containerMap: removing container" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453854 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefce3b9-edf9-402c-a781-2ad03e534677" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453866 4395 cpu_manager.go:414] "containerMap: removing container" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453870 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453883 4395 cpu_manager.go:414] "containerMap: removing container" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" Nov 01 22:31:41.453954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.453892 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" Nov 01 22:31:41.454243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.454000 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="31968d07-cb32-46ad-a888-d979e624c5e0" containerName="istio-proxy" Nov 01 22:31:41.454243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.454015 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a14cb7-9703-416a-9730-83950bb34b08" containerName="istio-proxy" Nov 01 22:31:41.454243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.454038 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6e0519-0259-4f2b-bb4e-9179ae59f022" containerName="istio-proxy" Nov 01 22:31:41.454243 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.454046 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da28f65-de84-410e-a58d-245c62b73a2a" containerName="revoke-keys" Nov 01 22:31:41.459128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.459110 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.462914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.462890 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:41.463109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.463005 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:41.466336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.466316 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7"] Nov 01 22:31:41.519299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.519267 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.519464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.519304 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.519464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.519338 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kr9\" (UniqueName: \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.621537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.621502 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.621537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.621538 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.621780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.621720 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kr9\" (UniqueName: \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.622231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.622208 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.624204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.624177 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.636517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.636486 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kr9\" (UniqueName: \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:41.775116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:41.775083 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:42.069948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:42.069920 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7"] Nov 01 22:31:42.071449 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:42.071419 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e7e38_6104_4e28_877d_cc0bb8f19122.slice/crio-bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989 WatchSource:0}: Error finding container bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989: Status 404 returned error can't find the container with id bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989 Nov 01 22:31:42.584449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:42.584412 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" event={"ID":"502e7e38-6104-4e28-877d-cc0bb8f19122","Type":"ContainerStarted","Data":"6d003b20f7da29c516dd7dc2d03a89691a983ae5eea7c0bdc911ebc331a0b31d"} Nov 01 22:31:42.584633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:42.584470 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" event={"ID":"502e7e38-6104-4e28-877d-cc0bb8f19122","Type":"ContainerStarted","Data":"bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989"} Nov 01 22:31:42.603345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:42.603291 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" podStartSLOduration=1.603277576 podStartE2EDuration="1.603277576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:42.602301324 +0000 UTC m=+2849.576203863" watchObservedRunningTime="2026-11-01 22:31:42.603277576 +0000 UTC m=+2849.577180114" Nov 01 22:31:42.944063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:42.943981 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:43.593078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:43.593004 4395 generic.go:356] "Generic (PLEG): container finished" podID="502e7e38-6104-4e28-877d-cc0bb8f19122" containerID="6d003b20f7da29c516dd7dc2d03a89691a983ae5eea7c0bdc911ebc331a0b31d" exitCode=0 Nov 01 22:31:43.593484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:43.593102 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" event={"ID":"502e7e38-6104-4e28-877d-cc0bb8f19122","Type":"ContainerDied","Data":"6d003b20f7da29c516dd7dc2d03a89691a983ae5eea7c0bdc911ebc331a0b31d"} Nov 01 22:31:44.761232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.761201 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:44.869893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.869798 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\") pod \"502e7e38-6104-4e28-877d-cc0bb8f19122\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " Nov 01 22:31:44.870112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.869903 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\" (UniqueName: \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\") pod \"502e7e38-6104-4e28-877d-cc0bb8f19122\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " Nov 01 22:31:44.870112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.870046 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\") pod \"502e7e38-6104-4e28-877d-cc0bb8f19122\" (UID: \"502e7e38-6104-4e28-877d-cc0bb8f19122\") " Nov 01 22:31:44.870377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.870350 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca" pod "502e7e38-6104-4e28-877d-cc0bb8f19122" (UID: "502e7e38-6104-4e28-877d-cc0bb8f19122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:44.872007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.871978 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca" pod "502e7e38-6104-4e28-877d-cc0bb8f19122" (UID: "502e7e38-6104-4e28-877d-cc0bb8f19122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:44.872323 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.872300 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9" pod "502e7e38-6104-4e28-877d-cc0bb8f19122" (UID: "502e7e38-6104-4e28-877d-cc0bb8f19122"). InnerVolumeSpecName "kube-api-access-54kr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:44.971341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.971311 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:44.971341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.971336 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54kr9\" (UniqueName: \"kubernetes.io/projected/502e7e38-6104-4e28-877d-cc0bb8f19122-kube-api-access-54kr9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:44.971341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:44.971347 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502e7e38-6104-4e28-877d-cc0bb8f19122-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:45.217690 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:45.217660 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e7e38_6104_4e28_877d_cc0bb8f19122.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:31:45.593285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:45.593197 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb"] Nov 01 22:31:45.596168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:45.596142 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d2832c-zxqwb"] Nov 01 22:31:45.608919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:45.608893 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" Nov 01 22:31:45.608919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:45.608908 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7" event={"ID":"502e7e38-6104-4e28-877d-cc0bb8f19122","Type":"ContainerDied","Data":"bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989"} Nov 01 22:31:45.609106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:45.608960 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbed0344881573009cc86a3cd622447844bf008c1740b5d542648f9e0e9c1989" Nov 01 22:31:46.318418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.318392 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:31:46.339640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.339610 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:31:46.389250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389215 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389285 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389362 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389425 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389477 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.389625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389509 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.389625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389547 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.389625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389579 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389622 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389891 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389676 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.389891 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389754 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.389891 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389824 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.390090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389899 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.390090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389933 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.390090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389958 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.390090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.389993 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.390090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.390081 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\") pod \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\" (UID: \"7525e218-1750-4b26-8d94-6a2a2e7a84a2\") " Nov 01 22:31:46.390379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.390111 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\") pod \"8139486a-45ba-4f0b-91e7-d20c1deedd62\" (UID: \"8139486a-45ba-4f0b-91e7-d20c1deedd62\") " Nov 01 22:31:46.390585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.390558 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.390683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.390637 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391186 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391286 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391374 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391627 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391649 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391850 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.392181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.391858 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.392639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.392465 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.395286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.395259 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.395498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.395331 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.396406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.396380 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.397724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.397697 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.398998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.398976 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "kube-api-access-zqh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.399705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.399207 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "kube-api-access-xhpb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.400477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.400443 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token" pod "7525e218-1750-4b26-8d94-6a2a2e7a84a2" (UID: "7525e218-1750-4b26-8d94-6a2a2e7a84a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.400612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.400588 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.403765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.403744 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:31:46.404888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.404860 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo" pod "8139486a-45ba-4f0b-91e7-d20c1deedd62" (UID: "8139486a-45ba-4f0b-91e7-d20c1deedd62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.407310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.407289 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:31:46.491523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491491 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491534 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491571 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491597 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491620 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491664 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.491693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491694 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491734 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491778 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491811 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491848 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491900 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491914 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.491982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\") pod \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\" (UID: \"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492014 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492067 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492087 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.492113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492118 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492159 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492201 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492220 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492228 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492249 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\") pod \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\" (UID: \"b28ee681-0fd2-4f75-8cd4-5358aea3f50d\") " Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492330 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.492664 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492495 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492679 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492918 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492934 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492940 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492956 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492973 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.492991 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493008 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493059 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493076 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493091 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493105 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493121 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493136 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493151 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhpb8\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-kube-api-access-xhpb8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493167 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493180 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493196 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139486a-45ba-4f0b-91e7-d20c1deedd62-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493216 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493237 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.493150 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493250 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493265 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493279 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7525e218-1750-4b26-8d94-6a2a2e7a84a2-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493295 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493308 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493321 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139486a-45ba-4f0b-91e7-d20c1deedd62-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493336 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139486a-45ba-4f0b-91e7-d20c1deedd62-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.494387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.493351 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqh4d\" (UniqueName: \"kubernetes.io/projected/7525e218-1750-4b26-8d94-6a2a2e7a84a2-kube-api-access-zqh4d\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.496125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496100 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.496242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496192 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "kube-api-access-cpfml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.496339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496320 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.496430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496412 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.496481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496433 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p" pod "b28ee681-0fd2-4f75-8cd4-5358aea3f50d" (UID: "b28ee681-0fd2-4f75-8cd4-5358aea3f50d"). InnerVolumeSpecName "kube-api-access-chk8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.496670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.496652 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:46.497307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.497281 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:46.497713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.497687 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo" pod "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" (UID: "d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594700 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594745 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594758 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594771 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594783 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.594807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594796 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chk8p\" (UniqueName: \"kubernetes.io/projected/b28ee681-0fd2-4f75-8cd4-5358aea3f50d-kube-api-access-chk8p\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.595171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594813 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfml\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-kube-api-access-cpfml\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.595171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594827 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.595171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.594839 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:46.617400 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.617363 4395 generic.go:356] "Generic (PLEG): container finished" podID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerID="54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0" exitCode=0 Nov 01 22:31:46.617567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.617420 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" Nov 01 22:31:46.617567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.617436 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" event={"ID":"8139486a-45ba-4f0b-91e7-d20c1deedd62","Type":"ContainerDied","Data":"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0"} Nov 01 22:31:46.617567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.617490 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t" event={"ID":"8139486a-45ba-4f0b-91e7-d20c1deedd62","Type":"ContainerDied","Data":"c834c7e1e079f0e0ae6224dfcf9343a9176361c8c383fab6b14c94eae3a15ba0"} Nov 01 22:31:46.617567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.617512 4395 scope.go:122] "RemoveContainer" containerID="54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0" Nov 01 22:31:46.619011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.618989 4395 generic.go:356] "Generic (PLEG): container finished" podID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerID="1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151" exitCode=0 Nov 01 22:31:46.619172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.619068 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" Nov 01 22:31:46.619172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.619096 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" event={"ID":"7525e218-1750-4b26-8d94-6a2a2e7a84a2","Type":"ContainerDied","Data":"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151"} Nov 01 22:31:46.619397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.619377 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b" event={"ID":"7525e218-1750-4b26-8d94-6a2a2e7a84a2","Type":"ContainerDied","Data":"19965f62c995347f72ba08dee6f303f88fd9fcdd2e11b6a615c68e45c9e3db34"} Nov 01 22:31:46.624367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.624340 4395 generic.go:356] "Generic (PLEG): container finished" podID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerID="c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920" exitCode=0 Nov 01 22:31:46.624518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.624381 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" Nov 01 22:31:46.624518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.624423 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" event={"ID":"b28ee681-0fd2-4f75-8cd4-5358aea3f50d","Type":"ContainerDied","Data":"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920"} Nov 01 22:31:46.624518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.624476 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl" event={"ID":"b28ee681-0fd2-4f75-8cd4-5358aea3f50d","Type":"ContainerDied","Data":"6d5451f7c9c9310ca36ed8150a92b397b0003a8ac39d70257ea5a3f1b80f0f5c"} Nov 01 22:31:46.626286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.626258 4395 generic.go:356] "Generic (PLEG): container finished" podID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerID="00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c" exitCode=0 Nov 01 22:31:46.626429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.626305 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" Nov 01 22:31:46.626429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.626330 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" event={"ID":"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1","Type":"ContainerDied","Data":"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c"} Nov 01 22:31:46.626429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.626372 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw" event={"ID":"d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1","Type":"ContainerDied","Data":"2fa18c74fc0bffc80bd6f8befe0fe2af55fdfc5f92474922d7deeb61920ecdf2"} Nov 01 22:31:46.629887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.629869 4395 scope.go:122] "RemoveContainer" containerID="54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0" Nov 01 22:31:46.630279 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:46.630253 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0\": container with ID starting with 54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0 not found: ID does not exist" containerID="54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0" Nov 01 22:31:46.630363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.630291 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0"} err="failed to get container status \"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0\": rpc error: code = NotFound desc = could not find container \"54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0\": container with ID starting with 54cafe0e97ad224f21f0e354af49fffd206e4daa4016cfe1f89b6e9a7126ead0 not found: ID does not exist" Nov 01 22:31:46.630363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.630318 4395 scope.go:122] "RemoveContainer" containerID="1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151" Nov 01 22:31:46.641729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.641710 4395 scope.go:122] "RemoveContainer" containerID="1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151" Nov 01 22:31:46.642010 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:46.641992 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151\": container with ID starting with 1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151 not found: ID does not exist" containerID="1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151" Nov 01 22:31:46.642078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.642019 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151"} err="failed to get container status \"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151\": rpc error: code = NotFound desc = could not find container \"1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151\": container with ID starting with 1bc4a30f6dc92611f83f1756a18b1b0e13470b74cd6b545e01072c31a4ea1151 not found: ID does not exist" Nov 01 22:31:46.642078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.642049 4395 scope.go:122] "RemoveContainer" containerID="c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920" Nov 01 22:31:46.653670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.653648 4395 scope.go:122] "RemoveContainer" containerID="c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920" Nov 01 22:31:46.653839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.653816 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:31:46.654012 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:46.653993 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920\": container with ID starting with c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920 not found: ID does not exist" containerID="c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920" Nov 01 22:31:46.654103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.654021 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920"} err="failed to get container status \"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920\": rpc error: code = NotFound desc = could not find container \"c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920\": container with ID starting with c161019864ad5e088217bd6c9e8517c5e6a610b6b6d27813d98de21e1e06d920 not found: ID does not exist" Nov 01 22:31:46.654103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.654061 4395 scope.go:122] "RemoveContainer" containerID="00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c" Nov 01 22:31:46.657291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.657272 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j9c9b"] Nov 01 22:31:46.664377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.664359 4395 scope.go:122] "RemoveContainer" containerID="00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c" Nov 01 22:31:46.664685 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:46.664666 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c\": container with ID starting with 00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c not found: ID does not exist" containerID="00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c" Nov 01 22:31:46.664757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.664692 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c"} err="failed to get container status \"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c\": rpc error: code = NotFound desc = could not find container \"00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c\": container with ID starting with 00a913d0a61f418ccc658133218b4d63804eb580f7f92ed39dedc5a2bac1e43c not found: ID does not exist" Nov 01 22:31:46.668576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.668553 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:31:46.672245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.672223 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-bzr2t"] Nov 01 22:31:46.684874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.684849 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:31:46.689108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.689082 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-j7cnw"] Nov 01 22:31:46.700862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.700839 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:31:46.706504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:46.706480 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-cv5vl"] Nov 01 22:31:47.165490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:47.165460 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da28f65-de84-410e-a58d-245c62b73a2a" path="/var/lib/kubelet/pods/3da28f65-de84-410e-a58d-245c62b73a2a/volumes" Nov 01 22:31:47.165911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:47.165897 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" path="/var/lib/kubelet/pods/7525e218-1750-4b26-8d94-6a2a2e7a84a2/volumes" Nov 01 22:31:47.166314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:47.166301 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" path="/var/lib/kubelet/pods/8139486a-45ba-4f0b-91e7-d20c1deedd62/volumes" Nov 01 22:31:47.167009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:47.166994 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" path="/var/lib/kubelet/pods/b28ee681-0fd2-4f75-8cd4-5358aea3f50d/volumes" Nov 01 22:31:47.167427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:47.167413 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" path="/var/lib/kubelet/pods/d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1/volumes" Nov 01 22:31:49.993635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:49.993608 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:31:50.024223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.024191 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:31:50.034538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.034514 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\" (UniqueName: \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\") pod \"e991dc18-a776-41e8-83b9-a2b9b4af510f\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " Nov 01 22:31:50.034700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.034685 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\") pod \"e991dc18-a776-41e8-83b9-a2b9b4af510f\" (UID: \"e991dc18-a776-41e8-83b9-a2b9b4af510f\") " Nov 01 22:31:50.035187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.035126 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume" pod "e991dc18-a776-41e8-83b9-a2b9b4af510f" (UID: "e991dc18-a776-41e8-83b9-a2b9b4af510f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:50.036742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.036718 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk" pod "e991dc18-a776-41e8-83b9-a2b9b4af510f" (UID: "e991dc18-a776-41e8-83b9-a2b9b4af510f"). InnerVolumeSpecName "kube-api-access-fdgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:50.135573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.135540 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\" (UniqueName: \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\") pod \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " Nov 01 22:31:50.135756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.135700 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\") pod \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\" (UID: \"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba\") " Nov 01 22:31:50.136095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.136063 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume" pod "ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" (UID: "ca67a24e-6d5f-4aca-9941-dc702d7cd3ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:50.136221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.136153 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdgpk\" (UniqueName: \"kubernetes.io/projected/e991dc18-a776-41e8-83b9-a2b9b4af510f-kube-api-access-fdgpk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:50.136221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.136172 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e991dc18-a776-41e8-83b9-a2b9b4af510f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:50.137931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.137910 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn" pod "ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" (UID: "ca67a24e-6d5f-4aca-9941-dc702d7cd3ba"). InnerVolumeSpecName "kube-api-access-p5pqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:50.237288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.237101 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5pqn\" (UniqueName: \"kubernetes.io/projected/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-kube-api-access-p5pqn\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:50.237288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.237134 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:50.664362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.664332 4395 generic.go:356] "Generic (PLEG): container finished" podID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerID="c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899" exitCode=0 Nov 01 22:31:50.664569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.664384 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" Nov 01 22:31:50.664569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.664417 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" event={"ID":"e991dc18-a776-41e8-83b9-a2b9b4af510f","Type":"ContainerDied","Data":"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899"} Nov 01 22:31:50.664569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.664468 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll" event={"ID":"e991dc18-a776-41e8-83b9-a2b9b4af510f","Type":"ContainerDied","Data":"7e3dbabf78ade69ebae4c768114ec636497b77f870e846b05da4fac859a1bb32"} Nov 01 22:31:50.664569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.664489 4395 scope.go:122] "RemoveContainer" containerID="c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899" Nov 01 22:31:50.666052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.665998 4395 generic.go:356] "Generic (PLEG): container finished" podID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerID="0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1" exitCode=0 Nov 01 22:31:50.666163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.666106 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" event={"ID":"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba","Type":"ContainerDied","Data":"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1"} Nov 01 22:31:50.666163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.666150 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" event={"ID":"ca67a24e-6d5f-4aca-9941-dc702d7cd3ba","Type":"ContainerDied","Data":"ed6ba829c911b4c0289b78d3125a8d4f6fdec5204eb6a38f614146c8db7f0adb"} Nov 01 22:31:50.666256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.666180 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq" Nov 01 22:31:50.676282 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.676258 4395 scope.go:122] "RemoveContainer" containerID="c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899" Nov 01 22:31:50.676598 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:50.676555 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899\": container with ID starting with c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899 not found: ID does not exist" containerID="c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899" Nov 01 22:31:50.676672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.676589 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899"} err="failed to get container status \"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899\": rpc error: code = NotFound desc = could not find container \"c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899\": container with ID starting with c10f1fdd82690deecb925dbc944d9a7e533e4fd7328645c61764a50bcf902899 not found: ID does not exist" Nov 01 22:31:50.676672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.676613 4395 scope.go:122] "RemoveContainer" containerID="0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1" Nov 01 22:31:50.685499 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.685481 4395 scope.go:122] "RemoveContainer" containerID="0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1" Nov 01 22:31:50.685766 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:50.685748 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1\": container with ID starting with 0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1 not found: ID does not exist" containerID="0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1" Nov 01 22:31:50.685815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.685773 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1"} err="failed to get container status \"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1\": rpc error: code = NotFound desc = could not find container \"0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1\": container with ID starting with 0ae8a64d8241c1ee0d614303e99ff494586a1558659c6812bd3d78670fc894d1 not found: ID does not exist" Nov 01 22:31:50.689540 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.689516 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:31:50.693772 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.693751 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-521734-867d896c58-kg8ll"] Nov 01 22:31:50.706688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.706666 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:31:50.711021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:50.711000 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-521734-f8998689f-p7gqq"] Nov 01 22:31:51.164774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:51.164745 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" path="/var/lib/kubelet/pods/ca67a24e-6d5f-4aca-9941-dc702d7cd3ba/volumes" Nov 01 22:31:51.165228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:51.165211 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" path="/var/lib/kubelet/pods/e991dc18-a776-41e8-83b9-a2b9b4af510f/volumes" Nov 01 22:31:51.580911 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:51.580875 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:31:52.234363 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:52.234326 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7"] Nov 01 22:31:52.237630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:52.237599 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e2bf63-vp6f7"] Nov 01 22:31:52.308615 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:52.308575 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:31:52.943482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:52.943450 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:31:53.164861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.164831 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502e7e38-6104-4e28-877d-cc0bb8f19122" path="/var/lib/kubelet/pods/502e7e38-6104-4e28-877d-cc0bb8f19122/volumes" Nov 01 22:31:53.920545 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.920507 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm"] Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.920993 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921006 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921038 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921043 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921053 4395 cpu_manager.go:414] "containerMap: removing container" podUID="502e7e38-6104-4e28-877d-cc0bb8f19122" containerName="revoke-keys" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921058 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502e7e38-6104-4e28-877d-cc0bb8f19122" containerName="revoke-keys" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921070 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921075 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" Nov 01 22:31:53.921077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921083 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921088 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921095 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921100 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921109 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921115 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921194 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8139486a-45ba-4f0b-91e7-d20c1deedd62" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921204 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991dc18-a776-41e8-83b9-a2b9b4af510f" containerName="payload-processing" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921211 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="502e7e38-6104-4e28-877d-cc0bb8f19122" containerName="revoke-keys" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921217 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7525e218-1750-4b26-8d94-6a2a2e7a84a2" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921224 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca67a24e-6d5f-4aca-9941-dc702d7cd3ba" containerName="payload-pre-processing" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921230 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18e56e6-4a7b-49cb-a8e5-6e340e32e8d1" containerName="istio-proxy" Nov 01 22:31:53.921611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.921236 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28ee681-0fd2-4f75-8cd4-5358aea3f50d" containerName="istio-proxy" Nov 01 22:31:53.925799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.925779 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:53.934700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.930550 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:31:53.934700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.930563 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:31:53.934700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.934642 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm"] Nov 01 22:31:53.991580 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.991549 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rjl\" (UniqueName: \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:53.991754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.991644 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:53.991857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:53.991837 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.092548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.092517 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rjl\" (UniqueName: \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.092738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.092588 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.092738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.092660 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.093325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.093302 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.095131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.095110 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.105074 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.105052 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rjl\" (UniqueName: \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\") pod \"maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.241901 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.241824 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:54.522637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.522612 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm"] Nov 01 22:31:54.524880 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:31:54.524849 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4744ad51_8368_47af_a28b_f37a1090e208.slice/crio-d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6 WatchSource:0}: Error finding container d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6: Status 404 returned error can't find the container with id d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6 Nov 01 22:31:54.693758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:54.693725 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" event={"ID":"4744ad51-8368-47af-a28b-f37a1090e208","Type":"ContainerStarted","Data":"d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6"} Nov 01 22:31:55.702073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:55.701966 4395 generic.go:356] "Generic (PLEG): container finished" podID="4744ad51-8368-47af-a28b-f37a1090e208" containerID="efcc54ac916594937cac8882cd5dd0d051fe76afb3bb5cc0730c554d48161960" exitCode=0 Nov 01 22:31:55.702073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:55.702056 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" event={"ID":"4744ad51-8368-47af-a28b-f37a1090e208","Type":"ContainerDied","Data":"efcc54ac916594937cac8882cd5dd0d051fe76afb3bb5cc0730c554d48161960"} Nov 01 22:31:56.582118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.582056 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" containerID="cri-o://b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77" gracePeriod=25 Nov 01 22:31:56.860721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.860697 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:56.927371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.927338 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\") pod \"4744ad51-8368-47af-a28b-f37a1090e208\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " Nov 01 22:31:56.927562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.927418 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\") pod \"4744ad51-8368-47af-a28b-f37a1090e208\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " Nov 01 22:31:56.927562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.927485 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\" (UniqueName: \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\") pod \"4744ad51-8368-47af-a28b-f37a1090e208\" (UID: \"4744ad51-8368-47af-a28b-f37a1090e208\") " Nov 01 22:31:56.927823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.927798 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca" pod "4744ad51-8368-47af-a28b-f37a1090e208" (UID: "4744ad51-8368-47af-a28b-f37a1090e208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:56.929701 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.929674 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca" pod "4744ad51-8368-47af-a28b-f37a1090e208" (UID: "4744ad51-8368-47af-a28b-f37a1090e208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:56.929827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:56.929741 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl" pod "4744ad51-8368-47af-a28b-f37a1090e208" (UID: "4744ad51-8368-47af-a28b-f37a1090e208"). InnerVolumeSpecName "kube-api-access-x9rjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:57.028953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.028922 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:57.028953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.028950 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4744ad51-8368-47af-a28b-f37a1090e208-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:57.028953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.028961 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rjl\" (UniqueName: \"kubernetes.io/projected/4744ad51-8368-47af-a28b-f37a1090e208-kube-api-access-x9rjl\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:57.277599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.277574 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:31:57.333009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.332976 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\" (UniqueName: \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\") pod \"13131b61-d3c7-4702-9d42-32469767084f\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " Nov 01 22:31:57.333207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.333114 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\") pod \"13131b61-d3c7-4702-9d42-32469767084f\" (UID: \"13131b61-d3c7-4702-9d42-32469767084f\") " Nov 01 22:31:57.333557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.333531 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume" pod "13131b61-d3c7-4702-9d42-32469767084f" (UID: "13131b61-d3c7-4702-9d42-32469767084f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:57.335345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.335321 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx" pod "13131b61-d3c7-4702-9d42-32469767084f" (UID: "13131b61-d3c7-4702-9d42-32469767084f"). InnerVolumeSpecName "kube-api-access-dvzwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:57.434181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.434108 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzwx\" (UniqueName: \"kubernetes.io/projected/13131b61-d3c7-4702-9d42-32469767084f-kube-api-access-dvzwx\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:57.434181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.434133 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13131b61-d3c7-4702-9d42-32469767084f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:57.718657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.718571 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" Nov 01 22:31:57.718657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.718636 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm" event={"ID":"4744ad51-8368-47af-a28b-f37a1090e208","Type":"ContainerDied","Data":"d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6"} Nov 01 22:31:57.718906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.718694 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92279f310987b156ae3e864ea191240ed9fffb71e759835955ca263668526b6" Nov 01 22:31:57.720294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.720266 4395 generic.go:356] "Generic (PLEG): container finished" podID="13131b61-d3c7-4702-9d42-32469767084f" containerID="1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658" exitCode=0 Nov 01 22:31:57.720431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.720305 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" Nov 01 22:31:57.720431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.720310 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" event={"ID":"13131b61-d3c7-4702-9d42-32469767084f","Type":"ContainerDied","Data":"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658"} Nov 01 22:31:57.720431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.720347 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2" event={"ID":"13131b61-d3c7-4702-9d42-32469767084f","Type":"ContainerDied","Data":"ddbe827858849a9bb90d2598b6ae1521b199c5d434fe033557511a2f251c2863"} Nov 01 22:31:57.720431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.720368 4395 scope.go:122] "RemoveContainer" containerID="1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658" Nov 01 22:31:57.730661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.730638 4395 scope.go:122] "RemoveContainer" containerID="1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658" Nov 01 22:31:57.730946 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:31:57.730925 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658\": container with ID starting with 1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658 not found: ID does not exist" containerID="1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658" Nov 01 22:31:57.731017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.730953 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658"} err="failed to get container status \"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658\": rpc error: code = NotFound desc = could not find container \"1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658\": container with ID starting with 1c6802e237d957982d44a5cf0a850b006254f25d105fd42f252233c0deebc658 not found: ID does not exist" Nov 01 22:31:57.745131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.745100 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:31:57.752191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:57.752166 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-1783e3-58446fb578-ddtf2"] Nov 01 22:31:58.752606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:58.752569 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 22:31:58.761504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:58.761471 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 22:31:59.164760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:31:59.164686 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13131b61-d3c7-4702-9d42-32469767084f" path="/var/lib/kubelet/pods/13131b61-d3c7-4702-9d42-32469767084f/volumes" Nov 01 22:32:00.107980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.107946 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:32:00.108530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.108488 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" containerID="cri-o://5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4" gracePeriod=30 Nov 01 22:32:00.112109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.112060 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:32:00.112334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.112307 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" containerID="cri-o://1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa" gracePeriod=30 Nov 01 22:32:00.715674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.715641 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g"] Nov 01 22:32:00.716154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716141 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4744ad51-8368-47af-a28b-f37a1090e208" containerName="revoke-keys" Nov 01 22:32:00.716208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716156 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4744ad51-8368-47af-a28b-f37a1090e208" containerName="revoke-keys" Nov 01 22:32:00.716208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716176 4395 cpu_manager.go:414] "containerMap: removing container" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" Nov 01 22:32:00.716208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716181 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" Nov 01 22:32:00.716299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716268 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4744ad51-8368-47af-a28b-f37a1090e208" containerName="revoke-keys" Nov 01 22:32:00.716299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.716280 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="13131b61-d3c7-4702-9d42-32469767084f" containerName="payload-processing" Nov 01 22:32:00.720917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.720898 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:00.731807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.731781 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g"] Nov 01 22:32:00.827244 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.827211 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:00.827448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.827350 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:00.904100 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.904065 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:00.904286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.904139 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wrf\" (UniqueName: \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:00.904286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:00.904245 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.006986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.006934 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.007246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.007045 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wrf\" (UniqueName: \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.007246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.007088 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.008392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.008366 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.010902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.010870 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.020623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.020592 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wrf\" (UniqueName: \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\") pod \"maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.035463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.035435 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:01.333167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.333144 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g"] Nov 01 22:32:01.334447 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:01.334419 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473821a_8917_4a6e_9a60_a6d4854d1224.slice/crio-f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9 WatchSource:0}: Error finding container f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9: Status 404 returned error can't find the container with id f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9 Nov 01 22:32:01.748861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.748824 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" event={"ID":"a473821a-8917-4a6e-9a60-a6d4854d1224","Type":"ContainerStarted","Data":"88e958e163d6fe345e8d9ef4f488040df15d10aa2322363e086eb2fff5a23b3b"} Nov 01 22:32:01.749083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.748881 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" event={"ID":"a473821a-8917-4a6e-9a60-a6d4854d1224","Type":"ContainerStarted","Data":"f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9"} Nov 01 22:32:01.768648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:01.768596 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" podStartSLOduration=1.7685823630000002 podStartE2EDuration="1.768582363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:01.766570635 +0000 UTC m=+2868.740473173" watchObservedRunningTime="2026-11-01 22:32:01.768582363 +0000 UTC m=+2868.742484901" Nov 01 22:32:02.489835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:02.489755 4395 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:32:02.489835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:02.489813 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:32:02.757092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:02.757012 4395 generic.go:356] "Generic (PLEG): container finished" podID="a473821a-8917-4a6e-9a60-a6d4854d1224" containerID="88e958e163d6fe345e8d9ef4f488040df15d10aa2322363e086eb2fff5a23b3b" exitCode=0 Nov 01 22:32:02.757262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:02.757100 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" event={"ID":"a473821a-8917-4a6e-9a60-a6d4854d1224","Type":"ContainerDied","Data":"88e958e163d6fe345e8d9ef4f488040df15d10aa2322363e086eb2fff5a23b3b"} Nov 01 22:32:02.886473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:02.886441 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:03.907805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.907771 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:03.942377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.942340 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\") pod \"a473821a-8917-4a6e-9a60-a6d4854d1224\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " Nov 01 22:32:03.942571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.942443 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\") pod \"a473821a-8917-4a6e-9a60-a6d4854d1224\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " Nov 01 22:32:03.942707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.942587 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\" (UniqueName: \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\") pod \"a473821a-8917-4a6e-9a60-a6d4854d1224\" (UID: \"a473821a-8917-4a6e-9a60-a6d4854d1224\") " Nov 01 22:32:03.943240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.943192 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca" pod "a473821a-8917-4a6e-9a60-a6d4854d1224" (UID: "a473821a-8917-4a6e-9a60-a6d4854d1224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:03.944941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.944912 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca" pod "a473821a-8917-4a6e-9a60-a6d4854d1224" (UID: "a473821a-8917-4a6e-9a60-a6d4854d1224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:03.945161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:03.945143 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf" pod "a473821a-8917-4a6e-9a60-a6d4854d1224" (UID: "a473821a-8917-4a6e-9a60-a6d4854d1224"). InnerVolumeSpecName "kube-api-access-j9wrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:04.044056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.043961 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:04.044056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.043991 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473821a-8917-4a6e-9a60-a6d4854d1224-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:04.044056 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.044001 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wrf\" (UniqueName: \"kubernetes.io/projected/a473821a-8917-4a6e-9a60-a6d4854d1224-kube-api-access-j9wrf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:04.082133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.082098 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 22:32:04.769625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.769593 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" Nov 01 22:32:04.769625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.769616 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g" event={"ID":"a473821a-8917-4a6e-9a60-a6d4854d1224","Type":"ContainerDied","Data":"f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9"} Nov 01 22:32:04.769841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:04.769667 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f401bf0ef5da585935ed5caaa10d9a63624b5c66d57953326c9d65a3fd9dbde9" Nov 01 22:32:05.416640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.416484 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:32:05.427416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.427392 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:32:05.462330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462300 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.462672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462351 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.462672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462406 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.462672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462462 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462843 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.462991 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463086 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463126 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463158 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463247 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463278 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463463 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.463720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463640 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.463720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463706 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463734 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463813 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.463902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463854 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.463902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463868 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.464178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463908 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.464178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463963 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\") pod \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\" (UID: \"168a20e7-07dd-49e4-a8ce-64fcbf16938f\") " Nov 01 22:32:05.464178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.463959 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.464178 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464057 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\") pod \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\" (UID: \"1b84cb7c-b176-4705-aa31-9f29cb0d650f\") " Nov 01 22:32:05.464407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464329 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.464600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464561 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464758 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464883 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464928 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464943 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464957 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464970 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464983 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.464995 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.465083 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.465449 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.465374 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.466090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.465484 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.467249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.467206 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "kube-api-access-4nfrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.468109 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.468089 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.469347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.469309 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.469439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.469355 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "kube-api-access-l6wgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.469439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.469355 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:05.469575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.469441 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo" pod "168a20e7-07dd-49e4-a8ce-64fcbf16938f" (UID: "168a20e7-07dd-49e4-a8ce-64fcbf16938f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:05.471394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.471360 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:05.471394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.471381 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token" pod "1b84cb7c-b176-4705-aa31-9f29cb0d650f" (UID: "1b84cb7c-b176-4705-aa31-9f29cb0d650f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:05.566395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566363 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566395 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566410 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566422 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b84cb7c-b176-4705-aa31-9f29cb0d650f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566434 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566445 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566457 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566469 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wgs\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-kube-api-access-l6wgs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566481 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168a20e7-07dd-49e4-a8ce-64fcbf16938f-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566494 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nfrk\" (UniqueName: \"kubernetes.io/projected/168a20e7-07dd-49e4-a8ce-64fcbf16938f-kube-api-access-4nfrk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.566610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.566507 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b84cb7c-b176-4705-aa31-9f29cb0d650f-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:05.776832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.776795 4395 generic.go:356] "Generic (PLEG): container finished" podID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerID="1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa" exitCode=0 Nov 01 22:32:05.777063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.776849 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" Nov 01 22:32:05.777063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.776880 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" event={"ID":"168a20e7-07dd-49e4-a8ce-64fcbf16938f","Type":"ContainerDied","Data":"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa"} Nov 01 22:32:05.777063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.776932 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz" event={"ID":"168a20e7-07dd-49e4-a8ce-64fcbf16938f","Type":"ContainerDied","Data":"7e4cc196360fe3ed0d588b4379928cd724d83ba879352faf4c9645513e9d0e54"} Nov 01 22:32:05.777063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.776958 4395 scope.go:122] "RemoveContainer" containerID="1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa" Nov 01 22:32:05.778696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.778549 4395 generic.go:356] "Generic (PLEG): container finished" podID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerID="5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4" exitCode=0 Nov 01 22:32:05.778696 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.778616 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" Nov 01 22:32:05.778877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.778750 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" event={"ID":"1b84cb7c-b176-4705-aa31-9f29cb0d650f","Type":"ContainerDied","Data":"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4"} Nov 01 22:32:05.778877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.778807 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw" event={"ID":"1b84cb7c-b176-4705-aa31-9f29cb0d650f","Type":"ContainerDied","Data":"34fde81a07a764d3e0567b02cf6bc3299d294c412ec1383e6554f4707c588405"} Nov 01 22:32:05.788407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.788352 4395 scope.go:122] "RemoveContainer" containerID="1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa" Nov 01 22:32:05.788654 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:05.788636 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa\": container with ID starting with 1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa not found: ID does not exist" containerID="1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa" Nov 01 22:32:05.788721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.788662 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa"} err="failed to get container status \"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa\": rpc error: code = NotFound desc = could not find container \"1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa\": container with ID starting with 1bfd2f164bd3be2fa6b595c95350b1505720f5917d832a20d1a85d67575956aa not found: ID does not exist" Nov 01 22:32:05.788721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.788684 4395 scope.go:122] "RemoveContainer" containerID="5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4" Nov 01 22:32:05.797415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.797396 4395 scope.go:122] "RemoveContainer" containerID="5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4" Nov 01 22:32:05.797687 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:05.797671 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4\": container with ID starting with 5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4 not found: ID does not exist" containerID="5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4" Nov 01 22:32:05.797730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.797694 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4"} err="failed to get container status \"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4\": rpc error: code = NotFound desc = could not find container \"5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4\": container with ID starting with 5dd87b1da0b19089a1fb42cdfde54136667aa7111034dd6bf8c6c50dda6cb4e4 not found: ID does not exist" Nov 01 22:32:05.802309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.802286 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:32:05.805571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.805550 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-ztrgz"] Nov 01 22:32:05.819174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.819150 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:32:05.823498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:05.823478 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-qprrw"] Nov 01 22:32:06.624568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.624495 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8"] Nov 01 22:32:06.625317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625287 4395 cpu_manager.go:414] "containerMap: removing container" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" Nov 01 22:32:06.625317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625315 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" Nov 01 22:32:06.625542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625344 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" Nov 01 22:32:06.625542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625352 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" Nov 01 22:32:06.625542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625367 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a473821a-8917-4a6e-9a60-a6d4854d1224" containerName="revoke-keys" Nov 01 22:32:06.625542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625374 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a473821a-8917-4a6e-9a60-a6d4854d1224" containerName="revoke-keys" Nov 01 22:32:06.625542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625527 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" containerName="istio-proxy" Nov 01 22:32:06.625714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625545 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" containerName="istio-proxy" Nov 01 22:32:06.625714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.625558 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a473821a-8917-4a6e-9a60-a6d4854d1224" containerName="revoke-keys" Nov 01 22:32:06.630155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.630135 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62"] Nov 01 22:32:06.630302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.630280 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.633755 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.633736 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.637957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.637887 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:06.637957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.637927 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:06.640997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.639622 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8"] Nov 01 22:32:06.642946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.642923 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62"] Nov 01 22:32:06.677346 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677322 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.677471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677375 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.677516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677506 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshkk\" (UniqueName: \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.677560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677552 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.677662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677622 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.677732 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.677716 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.778783 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.778749 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.778967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.778796 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.778967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.778825 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.778967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.778875 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.778967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.778928 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.779298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.779076 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshkk\" (UniqueName: \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.779750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.779661 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.779750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.779755 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.781653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.781630 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.782317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.782296 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.791159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.791119 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshkk\" (UniqueName: \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\") pod \"maas-api-revoke-sub-5eb28f4a-dlj62\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:06.791472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.791453 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\") pod \"maas-api-revoke-sub-7a17322f-t4dc8\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.951304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.951225 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:06.964508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:06.964472 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:07.163881 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.163850 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168a20e7-07dd-49e4-a8ce-64fcbf16938f" path="/var/lib/kubelet/pods/168a20e7-07dd-49e4-a8ce-64fcbf16938f/volumes" Nov 01 22:32:07.164340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.164325 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b84cb7c-b176-4705-aa31-9f29cb0d650f" path="/var/lib/kubelet/pods/1b84cb7c-b176-4705-aa31-9f29cb0d650f/volumes" Nov 01 22:32:07.249192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.249136 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8"] Nov 01 22:32:07.251889 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:07.251862 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a9c9b9_5f31_499a_9ad9_3b4203fabce6.slice/crio-1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1 WatchSource:0}: Error finding container 1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1: Status 404 returned error can't find the container with id 1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1 Nov 01 22:32:07.266212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.263313 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62"] Nov 01 22:32:07.266691 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:07.266664 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd862bd7_be6a_43cc_9368_9220366b33ce.slice/crio-8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d WatchSource:0}: Error finding container 8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d: Status 404 returned error can't find the container with id 8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d Nov 01 22:32:07.798711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.798665 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" event={"ID":"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6","Type":"ContainerStarted","Data":"43ceed1ee365aaa978ae0f07ab091e3262405490248922e62a54af7fa6a3bb6b"} Nov 01 22:32:07.799258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.798786 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" event={"ID":"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6","Type":"ContainerStarted","Data":"1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1"} Nov 01 22:32:07.800437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.800406 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" event={"ID":"bd862bd7-be6a-43cc-9368-9220366b33ce","Type":"ContainerStarted","Data":"89c85ac4298d86e671f108a9e28c06060fd05657bda7a0f8f0ac6f59a1c2a79e"} Nov 01 22:32:07.800567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.800454 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" event={"ID":"bd862bd7-be6a-43cc-9368-9220366b33ce","Type":"ContainerStarted","Data":"8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d"} Nov 01 22:32:07.835554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.835442 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" podStartSLOduration=1.835421762 podStartE2EDuration="1.835421762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:07.828430173 +0000 UTC m=+2874.802332713" watchObservedRunningTime="2026-11-01 22:32:07.835421762 +0000 UTC m=+2874.809324303" Nov 01 22:32:07.843915 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.843887 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:32:07.844158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.844136 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" containerID="cri-o://60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e" gracePeriod=30 Nov 01 22:32:07.850577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:07.850519 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" podStartSLOduration=1.850502307 podStartE2EDuration="1.850502307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:07.849412023 +0000 UTC m=+2874.823314562" watchObservedRunningTime="2026-11-01 22:32:07.850502307 +0000 UTC m=+2874.824404854" Nov 01 22:32:08.809242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:08.809207 4395 generic.go:356] "Generic (PLEG): container finished" podID="e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" containerID="43ceed1ee365aaa978ae0f07ab091e3262405490248922e62a54af7fa6a3bb6b" exitCode=0 Nov 01 22:32:08.809728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:08.809302 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" event={"ID":"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6","Type":"ContainerDied","Data":"43ceed1ee365aaa978ae0f07ab091e3262405490248922e62a54af7fa6a3bb6b"} Nov 01 22:32:08.810976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:08.810956 4395 generic.go:356] "Generic (PLEG): container finished" podID="bd862bd7-be6a-43cc-9368-9220366b33ce" containerID="89c85ac4298d86e671f108a9e28c06060fd05657bda7a0f8f0ac6f59a1c2a79e" exitCode=0 Nov 01 22:32:08.811101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:08.811059 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" event={"ID":"bd862bd7-be6a-43cc-9368-9220366b33ce","Type":"ContainerDied","Data":"89c85ac4298d86e671f108a9e28c06060fd05657bda7a0f8f0ac6f59a1c2a79e"} Nov 01 22:32:09.082821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:09.082731 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" containerID="cri-o://5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2" gracePeriod=25 Nov 01 22:32:09.991253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:09.991227 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:09.994438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:09.994413 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:10.030967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.030933 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\") pod \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " Nov 01 22:32:10.031122 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.031062 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\") pod \"bd862bd7-be6a-43cc-9368-9220366b33ce\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " Nov 01 22:32:10.031211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.031120 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\") pod \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " Nov 01 22:32:10.031211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.031203 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\" (UniqueName: \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\") pod \"bd862bd7-be6a-43cc-9368-9220366b33ce\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " Nov 01 22:32:10.031333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.031264 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\") pod \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\" (UID: \"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6\") " Nov 01 22:32:10.031333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.031297 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\") pod \"bd862bd7-be6a-43cc-9368-9220366b33ce\" (UID: \"bd862bd7-be6a-43cc-9368-9220366b33ce\") " Nov 01 22:32:10.032587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.032559 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca" pod "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" (UID: "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:10.032734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.032702 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca" pod "bd862bd7-be6a-43cc-9368-9220366b33ce" (UID: "bd862bd7-be6a-43cc-9368-9220366b33ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:10.036002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.035975 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca" pod "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" (UID: "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:10.036136 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.036003 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89" pod "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" (UID: "e1a9c9b9-5f31-499a-9ad9-3b4203fabce6"). InnerVolumeSpecName "kube-api-access-96f89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:10.037337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.037304 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk" pod "bd862bd7-be6a-43cc-9368-9220366b33ce" (UID: "bd862bd7-be6a-43cc-9368-9220366b33ce"). InnerVolumeSpecName "kube-api-access-cshkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:10.041350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.041327 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca" pod "bd862bd7-be6a-43cc-9368-9220366b33ce" (UID: "bd862bd7-be6a-43cc-9368-9220366b33ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133115 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f89\" (UniqueName: \"kubernetes.io/projected/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-kube-api-access-96f89\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133152 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshkk\" (UniqueName: \"kubernetes.io/projected/bd862bd7-be6a-43cc-9368-9220366b33ce-kube-api-access-cshkk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133165 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133173 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133182 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.133197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.133191 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd862bd7-be6a-43cc-9368-9220366b33ce-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:10.826748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.826658 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" event={"ID":"e1a9c9b9-5f31-499a-9ad9-3b4203fabce6","Type":"ContainerDied","Data":"1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1"} Nov 01 22:32:10.826748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.826712 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4f6cd7eb4951bb90fa7dcbfff07631a805ae190a222a70f72349e8a4e201f1" Nov 01 22:32:10.826977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.826791 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8" Nov 01 22:32:10.827513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.827487 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 22:32:10.829731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.829702 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" Nov 01 22:32:10.829731 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.829716 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62" event={"ID":"bd862bd7-be6a-43cc-9368-9220366b33ce","Type":"ContainerDied","Data":"8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d"} Nov 01 22:32:10.829929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.829763 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8c08c4994aa5b3e32704f2b8bb84c24ae92d68eaa11b93aace768d42135f8d" Nov 01 22:32:10.872106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:10.860045 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 22:32:12.336903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.336868 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm"] Nov 01 22:32:12.337345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337321 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" containerName="revoke-keys" Nov 01 22:32:12.337345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337332 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" containerName="revoke-keys" Nov 01 22:32:12.337428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337351 4395 cpu_manager.go:414] "containerMap: removing container" podUID="bd862bd7-be6a-43cc-9368-9220366b33ce" containerName="revoke-keys" Nov 01 22:32:12.337428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337356 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd862bd7-be6a-43cc-9368-9220366b33ce" containerName="revoke-keys" Nov 01 22:32:12.337428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337422 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" containerName="revoke-keys" Nov 01 22:32:12.337531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.337433 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd862bd7-be6a-43cc-9368-9220366b33ce" containerName="revoke-keys" Nov 01 22:32:12.340580 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.340559 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.346708 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.346683 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq"] Nov 01 22:32:12.347234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.347207 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:12.347391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.347222 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:12.351078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.351056 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm"] Nov 01 22:32:12.351186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.351119 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.373565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.373534 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq"] Nov 01 22:32:12.457536 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457503 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzl9\" (UniqueName: \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.457685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457553 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.457685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457576 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.457757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457693 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.457757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.457857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.457838 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bsk\" (UniqueName: \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.559254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559220 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.559254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559258 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.559496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559293 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.559496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559464 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.559619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559546 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bsk\" (UniqueName: \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.559667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.559641 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzl9\" (UniqueName: \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.561017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.560983 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.561158 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.560986 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.562242 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.562224 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.562687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.562669 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.595936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.595690 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzl9\" (UniqueName: \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\") pod \"maas-api-revoke-sub-4e230933-5b2pm\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.596347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.596327 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bsk\" (UniqueName: \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\") pod \"maas-api-revoke-sub-abb6a478-kpwsq\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.668374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.668086 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:12.680261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.672466 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:12.886232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.886204 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:12.964918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.964885 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm"] Nov 01 22:32:12.968424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:12.968399 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq"] Nov 01 22:32:12.968524 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:12.968454 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ab6af0_91ca_4396_a1e1_fe588b28e3c5.slice/crio-ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749 WatchSource:0}: Error finding container ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749: Status 404 returned error can't find the container with id ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749 Nov 01 22:32:12.970336 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:12.970311 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe701dd7_3026_4129_970d_8a0dbca73b67.slice/crio-bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863 WatchSource:0}: Error finding container bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863: Status 404 returned error can't find the container with id bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863 Nov 01 22:32:13.197211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.197177 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:32:13.268666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268636 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.268857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268684 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.268857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268703 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.268857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268778 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.268857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268834 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.269130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268887 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.269130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268950 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.269130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268945 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:13.269130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.268986 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:13.269130 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269093 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.269385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269132 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\") pod \"0b36179e-0951-4274-baaa-b468ed45bffe\" (UID: \"0b36179e-0951-4274-baaa-b468ed45bffe\") " Nov 01 22:32:13.269385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269261 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:13.269385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269262 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:13.269510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269422 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:13.269666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269642 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.269756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269673 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.269756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269690 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.269756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269705 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b36179e-0951-4274-baaa-b468ed45bffe-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.270118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.269759 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.271634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.271603 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:13.271634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.271603 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:13.271958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.271939 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:13.272006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.271981 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd" pod "0b36179e-0951-4274-baaa-b468ed45bffe" (UID: "0b36179e-0951-4274-baaa-b468ed45bffe"). InnerVolumeSpecName "kube-api-access-k69hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:13.370870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.370835 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69hd\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-kube-api-access-k69hd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.370870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.370870 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b36179e-0951-4274-baaa-b468ed45bffe-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.371274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.370887 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b36179e-0951-4274-baaa-b468ed45bffe-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.371274 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.370900 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b36179e-0951-4274-baaa-b468ed45bffe-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:13.776736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.776667 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" containerID="cri-o://d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f" gracePeriod=45 Nov 01 22:32:13.784698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.784583 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" containerID="cri-o://8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9" gracePeriod=45 Nov 01 22:32:13.850173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.850065 4395 generic.go:356] "Generic (PLEG): container finished" podID="0b36179e-0951-4274-baaa-b468ed45bffe" containerID="60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e" exitCode=0 Nov 01 22:32:13.850173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.850117 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" Nov 01 22:32:13.850427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.850151 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" event={"ID":"0b36179e-0951-4274-baaa-b468ed45bffe","Type":"ContainerDied","Data":"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e"} Nov 01 22:32:13.850565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.850509 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr" event={"ID":"0b36179e-0951-4274-baaa-b468ed45bffe","Type":"ContainerDied","Data":"38b74e739d370ee1528260f33e4ceebd963ab27911b4804d846ac4cf684f2bb2"} Nov 01 22:32:13.850647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.850572 4395 scope.go:122] "RemoveContainer" containerID="60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e" Nov 01 22:32:13.853562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.853076 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" event={"ID":"22ab6af0-91ca-4396-a1e1-fe588b28e3c5","Type":"ContainerStarted","Data":"c2b31227dedebf209f2e1bcfa4800e024d1604389a808b1ae287933cd287e2e0"} Nov 01 22:32:13.853858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.853835 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" event={"ID":"22ab6af0-91ca-4396-a1e1-fe588b28e3c5","Type":"ContainerStarted","Data":"ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749"} Nov 01 22:32:13.855124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.855100 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" event={"ID":"be701dd7-3026-4129-970d-8a0dbca73b67","Type":"ContainerStarted","Data":"450cb827536e1a4991b7567a0ed87ade82ed1405dee3e75206578b3c3275b43c"} Nov 01 22:32:13.855200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.855139 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" event={"ID":"be701dd7-3026-4129-970d-8a0dbca73b67","Type":"ContainerStarted","Data":"bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863"} Nov 01 22:32:13.872763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.872721 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" podStartSLOduration=1.872708905 podStartE2EDuration="1.872708905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:13.870653721 +0000 UTC m=+2880.844556275" watchObservedRunningTime="2026-11-01 22:32:13.872708905 +0000 UTC m=+2880.846611690" Nov 01 22:32:13.889505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.889467 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" podStartSLOduration=1.889455684 podStartE2EDuration="1.889455684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:13.887165493 +0000 UTC m=+2880.861068032" watchObservedRunningTime="2026-11-01 22:32:13.889455684 +0000 UTC m=+2880.863358244" Nov 01 22:32:13.910503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.910462 4395 scope.go:122] "RemoveContainer" containerID="60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e" Nov 01 22:32:13.910878 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:13.910846 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e\": container with ID starting with 60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e not found: ID does not exist" containerID="60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e" Nov 01 22:32:13.910955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.910897 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e"} err="failed to get container status \"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e\": rpc error: code = NotFound desc = could not find container \"60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e\": container with ID starting with 60e7ba256d8ec2815b4a90451eabec5c91889839456001c15fd1b150dd55988e not found: ID does not exist" Nov 01 22:32:13.921846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.921804 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:32:13.931064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:13.926155 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-qllnr"] Nov 01 22:32:14.131629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.131537 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:32:14.136715 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.136687 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:32:14.179391 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179357 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.179574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179402 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.179574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179437 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179463 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\" (UniqueName: \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179489 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179532 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\" (UniqueName: \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179593 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179620 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179680 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179752 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179782 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\") pod \"8640e002-c647-4ac9-813e-d93f131dfcb2\" (UID: \"8640e002-c647-4ac9-813e-d93f131dfcb2\") " Nov 01 22:32:14.179840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.179807 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\") pod \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\" (UID: \"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9\") " Nov 01 22:32:14.183859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.180390 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.184080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.183785 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.186663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.186458 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.186777 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.186736 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.186965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.186926 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.187061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.186967 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.188053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.187985 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.194468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.194052 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:14.194468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.194210 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "kube-api-access-wpzbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:14.194468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.194321 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:14.194468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.194410 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8" pod "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" (UID: "c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9"). InnerVolumeSpecName "kube-api-access-fs8c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:14.196410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.196382 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs" pod "8640e002-c647-4ac9-813e-d93f131dfcb2" (UID: "8640e002-c647-4ac9-813e-d93f131dfcb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:14.281637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281612 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281636 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8640e002-c647-4ac9-813e-d93f131dfcb2-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281645 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281654 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281662 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281670 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281678 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281687 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpzbz\" (UniqueName: \"kubernetes.io/projected/8640e002-c647-4ac9-813e-d93f131dfcb2-kube-api-access-wpzbz\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281696 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281704 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs8c8\" (UniqueName: \"kubernetes.io/projected/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-kube-api-access-fs8c8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281714 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.281838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.281721 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8640e002-c647-4ac9-813e-d93f131dfcb2-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:14.872832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.872735 4395 generic.go:356] "Generic (PLEG): container finished" podID="22ab6af0-91ca-4396-a1e1-fe588b28e3c5" containerID="c2b31227dedebf209f2e1bcfa4800e024d1604389a808b1ae287933cd287e2e0" exitCode=0 Nov 01 22:32:14.873326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.872821 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" event={"ID":"22ab6af0-91ca-4396-a1e1-fe588b28e3c5","Type":"ContainerDied","Data":"c2b31227dedebf209f2e1bcfa4800e024d1604389a808b1ae287933cd287e2e0"} Nov 01 22:32:14.874592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.874569 4395 generic.go:356] "Generic (PLEG): container finished" podID="be701dd7-3026-4129-970d-8a0dbca73b67" containerID="450cb827536e1a4991b7567a0ed87ade82ed1405dee3e75206578b3c3275b43c" exitCode=0 Nov 01 22:32:14.874757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.874654 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" event={"ID":"be701dd7-3026-4129-970d-8a0dbca73b67","Type":"ContainerDied","Data":"450cb827536e1a4991b7567a0ed87ade82ed1405dee3e75206578b3c3275b43c"} Nov 01 22:32:14.875948 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.875922 4395 generic.go:356] "Generic (PLEG): container finished" podID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerID="d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f" exitCode=0 Nov 01 22:32:14.876059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.875973 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" Nov 01 22:32:14.876059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.875983 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" event={"ID":"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9","Type":"ContainerDied","Data":"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f"} Nov 01 22:32:14.876059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.876004 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97" event={"ID":"c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9","Type":"ContainerDied","Data":"45abd67df1df39c7320b491f10f99b9bbaf18326a756aeb9d5d14bd3dd467870"} Nov 01 22:32:14.876059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.876019 4395 scope.go:122] "RemoveContainer" containerID="d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f" Nov 01 22:32:14.877517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.877494 4395 generic.go:356] "Generic (PLEG): container finished" podID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerID="8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9" exitCode=0 Nov 01 22:32:14.877626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.877522 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" event={"ID":"8640e002-c647-4ac9-813e-d93f131dfcb2","Type":"ContainerDied","Data":"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9"} Nov 01 22:32:14.877626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.877546 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" Nov 01 22:32:14.877626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.877558 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" event={"ID":"8640e002-c647-4ac9-813e-d93f131dfcb2","Type":"ContainerDied","Data":"2af3c14bdd8550eba9f02251e14529946f79b162cd28a6ff3b4f02dd9e6798a6"} Nov 01 22:32:14.887926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.887843 4395 scope.go:122] "RemoveContainer" containerID="d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f" Nov 01 22:32:14.888200 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:14.888186 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f\": container with ID starting with d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f not found: ID does not exist" containerID="d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f" Nov 01 22:32:14.888374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.888211 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f"} err="failed to get container status \"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f\": rpc error: code = NotFound desc = could not find container \"d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f\": container with ID starting with d51cd9c1581e434ce0b9171adcd62c73b792edd139ca63de9dc9b727f0c7c46f not found: ID does not exist" Nov 01 22:32:14.888374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.888226 4395 scope.go:122] "RemoveContainer" containerID="8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9" Nov 01 22:32:14.897179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.897161 4395 scope.go:122] "RemoveContainer" containerID="8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9" Nov 01 22:32:14.897475 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:14.897457 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9\": container with ID starting with 8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9 not found: ID does not exist" containerID="8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9" Nov 01 22:32:14.897532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.897480 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9"} err="failed to get container status \"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9\": rpc error: code = NotFound desc = could not find container \"8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9\": container with ID starting with 8e5eb36739a53d5276e5b1f60b3e46c27e0a1242e5fcc7113862d345ae2bfef9 not found: ID does not exist" Nov 01 22:32:14.908247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.908225 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 22:32:14.914638 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.914605 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849"] Nov 01 22:32:14.943268 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.943244 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 22:32:14.947981 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:14.947960 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ff3e70/distinct-w3-ff3e70-kserve-564f777f9c-cxk97"] Nov 01 22:32:15.065548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.065510 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ff3e70/premium-w3-ff3e70-kserve-6c6c494798-ss849" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:32:15.167287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.167204 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" path="/var/lib/kubelet/pods/0b36179e-0951-4274-baaa-b468ed45bffe/volumes" Nov 01 22:32:15.167744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.167723 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" path="/var/lib/kubelet/pods/8640e002-c647-4ac9-813e-d93f131dfcb2/volumes" Nov 01 22:32:15.168216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.168197 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" path="/var/lib/kubelet/pods/c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9/volumes" Nov 01 22:32:15.828073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.828014 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" containerID="cri-o://13dec8004fee61986a613c5ddab33d216bf61e2c051c4b98a1db8ccd5d33c15f" gracePeriod=25 Nov 01 22:32:15.861289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:15.861258 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" containerID="cri-o://ec83eab834fc5ee337c32b6e5a96de39c2008224c6ef25b0a8b9553185561202" gracePeriod=25 Nov 01 22:32:16.049401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.049372 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:16.051764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.051747 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:16.097275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.097201 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\" (UniqueName: \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\") pod \"be701dd7-3026-4129-970d-8a0dbca73b67\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " Nov 01 22:32:16.097428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.097304 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\") pod \"be701dd7-3026-4129-970d-8a0dbca73b67\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " Nov 01 22:32:16.097508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.097428 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\") pod \"be701dd7-3026-4129-970d-8a0dbca73b67\" (UID: \"be701dd7-3026-4129-970d-8a0dbca73b67\") " Nov 01 22:32:16.098283 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.098253 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca" pod "be701dd7-3026-4129-970d-8a0dbca73b67" (UID: "be701dd7-3026-4129-970d-8a0dbca73b67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:16.099599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.099575 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca" pod "be701dd7-3026-4129-970d-8a0dbca73b67" (UID: "be701dd7-3026-4129-970d-8a0dbca73b67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:16.099718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.099640 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk" pod "be701dd7-3026-4129-970d-8a0dbca73b67" (UID: "be701dd7-3026-4129-970d-8a0dbca73b67"). InnerVolumeSpecName "kube-api-access-c8bsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:16.198799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.198772 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\" (UniqueName: \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\") pod \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " Nov 01 22:32:16.198972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.198846 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\") pod \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " Nov 01 22:32:16.198972 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.198939 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\") pod \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\" (UID: \"22ab6af0-91ca-4396-a1e1-fe588b28e3c5\") " Nov 01 22:32:16.199264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.199225 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca" pod "22ab6af0-91ca-4396-a1e1-fe588b28e3c5" (UID: "22ab6af0-91ca-4396-a1e1-fe588b28e3c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:16.199264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.199260 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.199467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.199280 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be701dd7-3026-4129-970d-8a0dbca73b67-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.199467 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.199296 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bsk\" (UniqueName: \"kubernetes.io/projected/be701dd7-3026-4129-970d-8a0dbca73b67-kube-api-access-c8bsk\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.201358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.201326 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9" pod "22ab6af0-91ca-4396-a1e1-fe588b28e3c5" (UID: "22ab6af0-91ca-4396-a1e1-fe588b28e3c5"). InnerVolumeSpecName "kube-api-access-ntzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:16.201534 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.201507 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca" pod "22ab6af0-91ca-4396-a1e1-fe588b28e3c5" (UID: "22ab6af0-91ca-4396-a1e1-fe588b28e3c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:16.300491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.300459 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.300491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.300486 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntzl9\" (UniqueName: \"kubernetes.io/projected/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-kube-api-access-ntzl9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.300491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.300496 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22ab6af0-91ca-4396-a1e1-fe588b28e3c5-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.712712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.712673 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:16.838142 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.838083 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:32:16.894637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.894609 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" Nov 01 22:32:16.894637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.894629 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm" event={"ID":"22ab6af0-91ca-4396-a1e1-fe588b28e3c5","Type":"ContainerDied","Data":"ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749"} Nov 01 22:32:16.894829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.894675 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb2d0fd1bf88fb40e0c363ceac462d63ceef26e92193524ce99c10224950749" Nov 01 22:32:16.896135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.896118 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" Nov 01 22:32:16.896249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.896165 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq" event={"ID":"be701dd7-3026-4129-970d-8a0dbca73b67","Type":"ContainerDied","Data":"bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863"} Nov 01 22:32:16.896249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.896208 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb5bc5faec1b94c96cbe28addb7489b5c1019fd381240784c98adfe7b76ef863" Nov 01 22:32:16.897686 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.897664 4395 generic.go:356] "Generic (PLEG): container finished" podID="7f7804bb-aec3-4506-a386-008177911e3f" containerID="b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77" exitCode=0 Nov 01 22:32:16.897813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.897708 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" Nov 01 22:32:16.897813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.897748 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" event={"ID":"7f7804bb-aec3-4506-a386-008177911e3f","Type":"ContainerDied","Data":"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77"} Nov 01 22:32:16.897813 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.897808 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7" event={"ID":"7f7804bb-aec3-4506-a386-008177911e3f","Type":"ContainerDied","Data":"949a5b47d695df0ea61c643e9d1d8225f589117507ddda9b6d3caf393394efde"} Nov 01 22:32:16.898040 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.897839 4395 scope.go:122] "RemoveContainer" containerID="b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77" Nov 01 22:32:16.906870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.906848 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\") pod \"7f7804bb-aec3-4506-a386-008177911e3f\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " Nov 01 22:32:16.907003 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.906991 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\" (UniqueName: \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\") pod \"7f7804bb-aec3-4506-a386-008177911e3f\" (UID: \"7f7804bb-aec3-4506-a386-008177911e3f\") " Nov 01 22:32:16.907255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.907235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume" pod "7f7804bb-aec3-4506-a386-008177911e3f" (UID: "7f7804bb-aec3-4506-a386-008177911e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:16.907403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.907388 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7804bb-aec3-4506-a386-008177911e3f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:16.909146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.909125 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv" pod "7f7804bb-aec3-4506-a386-008177911e3f" (UID: "7f7804bb-aec3-4506-a386-008177911e3f"). InnerVolumeSpecName "kube-api-access-s7wmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:16.909194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.909151 4395 scope.go:122] "RemoveContainer" containerID="b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77" Nov 01 22:32:16.909459 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:16.909441 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77\": container with ID starting with b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77 not found: ID does not exist" containerID="b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77" Nov 01 22:32:16.909503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:16.909467 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77"} err="failed to get container status \"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77\": rpc error: code = NotFound desc = could not find container \"b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77\": container with ID starting with b9ee1365643622f08bd2ecc5bd463e957b8f86d9050a9cd73f07a3e87405bd77 not found: ID does not exist" Nov 01 22:32:17.008935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.008904 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wmv\" (UniqueName: \"kubernetes.io/projected/7f7804bb-aec3-4506-a386-008177911e3f-kube-api-access-s7wmv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:17.154474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.154402 4395 scope.go:122] "RemoveContainer" containerID="80410bbfa96e9f9668473e974381a707ce4874cdaa6ad3ae361e5f2926d80657" Nov 01 22:32:17.221091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.220970 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:32:17.224186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.224164 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-e2bf63-7f476f467f-bxjl7"] Nov 01 22:32:17.719980 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.719935 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:17.720190 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:17.720049 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:19.164574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:19.164539 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7804bb-aec3-4506-a386-008177911e3f" path="/var/lib/kubelet/pods/7f7804bb-aec3-4506-a386-008177911e3f/volumes" Nov 01 22:32:21.804834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:21.804799 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm"] Nov 01 22:32:21.813378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:21.813347 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ff3e70-8hrpm"] Nov 01 22:32:22.312800 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:22.312740 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:32:22.860823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:22.860787 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g"] Nov 01 22:32:22.870611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:22.870573 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0def36-6hz4g"] Nov 01 22:32:23.165125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:23.165054 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4744ad51-8368-47af-a28b-f37a1090e208" path="/var/lib/kubelet/pods/4744ad51-8368-47af-a28b-f37a1090e208/volumes" Nov 01 22:32:23.165548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:23.165534 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a473821a-8917-4a6e-9a60-a6d4854d1224" path="/var/lib/kubelet/pods/a473821a-8917-4a6e-9a60-a6d4854d1224/volumes" Nov 01 22:32:26.712810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:26.712769 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:27.719835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:27.719795 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:27.719835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:27.719814 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:29.331196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.331173 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 22:32:29.450237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.450163 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\" (UniqueName: \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\") pod \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " Nov 01 22:32:29.450396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.450278 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\") pod \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\" (UID: \"b8b827fa-0b84-4c05-812b-50ddf3e8d181\") " Nov 01 22:32:29.450659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.450633 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume" pod "b8b827fa-0b84-4c05-812b-50ddf3e8d181" (UID: "b8b827fa-0b84-4c05-812b-50ddf3e8d181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:29.452478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.452449 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz" pod "b8b827fa-0b84-4c05-812b-50ddf3e8d181" (UID: "b8b827fa-0b84-4c05-812b-50ddf3e8d181"). InnerVolumeSpecName "kube-api-access-4jbcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:29.551069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.551019 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbcz\" (UniqueName: \"kubernetes.io/projected/b8b827fa-0b84-4c05-812b-50ddf3e8d181-kube-api-access-4jbcz\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:29.551069 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.551067 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b827fa-0b84-4c05-812b-50ddf3e8d181-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:29.694107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.687483 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 22:32:29.694107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.687748 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" containerID="cri-o://38c317f4265cf8edbb3d86a82f1ef79281436ce5bb17fce1f4389fd642f4b2ff" gracePeriod=30 Nov 01 22:32:29.985605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.985562 4395 generic.go:356] "Generic (PLEG): container finished" podID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerID="5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2" exitCode=0 Nov 01 22:32:29.985802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.985684 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" Nov 01 22:32:29.985802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.985706 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" event={"ID":"b8b827fa-0b84-4c05-812b-50ddf3e8d181","Type":"ContainerDied","Data":"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2"} Nov 01 22:32:29.985802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.985763 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r" event={"ID":"b8b827fa-0b84-4c05-812b-50ddf3e8d181","Type":"ContainerDied","Data":"224fbbab2322e97b84571451f4237a5c94f773f1c09fbdcacda6c419d3f36b0d"} Nov 01 22:32:29.985802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.985787 4395 scope.go:122] "RemoveContainer" containerID="5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2" Nov 01 22:32:29.997867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.997849 4395 scope.go:122] "RemoveContainer" containerID="5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2" Nov 01 22:32:29.998205 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:29.998186 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2\": container with ID starting with 5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2 not found: ID does not exist" containerID="5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2" Nov 01 22:32:29.998285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:29.998215 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2"} err="failed to get container status \"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2\": rpc error: code = NotFound desc = could not find container \"5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2\": container with ID starting with 5e462d1cb9c5361db37b49970d22eb2336af0719c1407b4c8e31d99b501878d2 not found: ID does not exist" Nov 01 22:32:30.013319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:30.013294 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 22:32:30.017787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:30.017764 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ff3e70-549bf5bb59-tsv9r"] Nov 01 22:32:31.100456 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:31.100415 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 22:32:31.100976 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:31.100665 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" containerID="cri-o://51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a" gracePeriod=30 Nov 01 22:32:31.164938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:31.164908 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" path="/var/lib/kubelet/pods/b8b827fa-0b84-4c05-812b-50ddf3e8d181/volumes" Nov 01 22:32:32.352348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:32.352309 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:32:32.353554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:32.353356 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:32:33.126956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.126922 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx"] Nov 01 22:32:33.127372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127358 4395 cpu_manager.go:414] "containerMap: removing container" podUID="22ab6af0-91ca-4396-a1e1-fe588b28e3c5" containerName="revoke-keys" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127373 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ab6af0-91ca-4396-a1e1-fe588b28e3c5" containerName="revoke-keys" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127383 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127390 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127398 4395 cpu_manager.go:414] "containerMap: removing container" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127403 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127412 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127416 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" Nov 01 22:32:33.127423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127423 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127428 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127436 4395 cpu_manager.go:414] "containerMap: removing container" podUID="be701dd7-3026-4129-970d-8a0dbca73b67" containerName="revoke-keys" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127440 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be701dd7-3026-4129-970d-8a0dbca73b67" containerName="revoke-keys" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127447 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127451 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127527 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b36179e-0951-4274-baaa-b468ed45bffe" containerName="istio-proxy" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127536 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61b9164-97c2-4eeb-ab5b-f44c4a7e08b9" containerName="main" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127544 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8640e002-c647-4ac9-813e-d93f131dfcb2" containerName="main" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127550 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="be701dd7-3026-4129-970d-8a0dbca73b67" containerName="revoke-keys" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127555 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b827fa-0b84-4c05-812b-50ddf3e8d181" containerName="payload-pre-processing" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127565 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ab6af0-91ca-4396-a1e1-fe588b28e3c5" containerName="revoke-keys" Nov 01 22:32:33.127676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.127571 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7804bb-aec3-4506-a386-008177911e3f" containerName="payload-pre-processing" Nov 01 22:32:33.132177 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.132155 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.139369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.139342 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx"] Nov 01 22:32:33.186616 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.186569 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:33.186796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.186668 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:33.191163 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.191135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.191377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.191328 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.191623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.191460 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sbj\" (UniqueName: \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.292878 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.292850 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.292991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.292909 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sbj\" (UniqueName: \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.292991 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.292979 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.293447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.293428 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.295558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.295538 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.305415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.305392 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sbj\" (UniqueName: \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.446958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.446860 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:33.724485 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:33.724462 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx"] Nov 01 22:32:33.726498 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:33.726471 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c931dd0_c825_477e_baf2_a17fef703ead.slice/crio-ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753 WatchSource:0}: Error finding container ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753: Status 404 returned error can't find the container with id ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753 Nov 01 22:32:34.011832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:34.011798 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" event={"ID":"9c931dd0-c825-477e-baf2-a17fef703ead","Type":"ContainerStarted","Data":"5d58f171f6dec4f36370b160d9182cd0419dcb26d4417eb4a837449b877584ff"} Nov 01 22:32:34.011954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:34.011858 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" event={"ID":"9c931dd0-c825-477e-baf2-a17fef703ead","Type":"ContainerStarted","Data":"ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753"} Nov 01 22:32:34.029141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:34.029103 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" podStartSLOduration=1.029089799 podStartE2EDuration="1.029089799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:34.028498645 +0000 UTC m=+2901.002401185" watchObservedRunningTime="2026-11-01 22:32:34.029089799 +0000 UTC m=+2901.002992337" Nov 01 22:32:35.018840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.018808 4395 generic.go:356] "Generic (PLEG): container finished" podID="9c931dd0-c825-477e-baf2-a17fef703ead" containerID="5d58f171f6dec4f36370b160d9182cd0419dcb26d4417eb4a837449b877584ff" exitCode=0 Nov 01 22:32:35.019290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.018851 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" event={"ID":"9c931dd0-c825-477e-baf2-a17fef703ead","Type":"ContainerDied","Data":"5d58f171f6dec4f36370b160d9182cd0419dcb26d4417eb4a837449b877584ff"} Nov 01 22:32:35.020921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.020894 4395 generic.go:356] "Generic (PLEG): container finished" podID="9f995556-a12a-4021-950a-7130a1467e7e" containerID="38c317f4265cf8edbb3d86a82f1ef79281436ce5bb17fce1f4389fd642f4b2ff" exitCode=0 Nov 01 22:32:35.021104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.020982 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" event={"ID":"9f995556-a12a-4021-950a-7130a1467e7e","Type":"ContainerDied","Data":"38c317f4265cf8edbb3d86a82f1ef79281436ce5bb17fce1f4389fd642f4b2ff"} Nov 01 22:32:35.368860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.368835 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 22:32:35.422127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422093 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422184 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422216 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422304 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422328 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422352 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422377 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422376 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.422636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422443 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.422636 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.422476 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\") pod \"9f995556-a12a-4021-950a-7130a1467e7e\" (UID: \"9f995556-a12a-4021-950a-7130a1467e7e\") " Nov 01 22:32:35.423290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.423215 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.423511 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.423405 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:35.423611 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.423574 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.423766 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.423746 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.423896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.423789 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.426072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.425083 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:35.426072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.425352 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:35.426072 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.425380 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "kube-api-access-jdwh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:35.426316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.426131 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy" pod "9f995556-a12a-4021-950a-7130a1467e7e" (UID: "9f995556-a12a-4021-950a-7130a1467e7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:35.525194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525162 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525187 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525199 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525207 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f995556-a12a-4021-950a-7130a1467e7e-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525216 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f995556-a12a-4021-950a-7130a1467e7e-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525224 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525232 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f995556-a12a-4021-950a-7130a1467e7e-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:35.525426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:35.525240 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwh9\" (UniqueName: \"kubernetes.io/projected/9f995556-a12a-4021-950a-7130a1467e7e-kube-api-access-jdwh9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.030674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.030641 4395 generic.go:356] "Generic (PLEG): container finished" podID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerID="13dec8004fee61986a613c5ddab33d216bf61e2c051c4b98a1db8ccd5d33c15f" exitCode=0 Nov 01 22:32:36.031153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.030756 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" event={"ID":"50de95f8-e75d-4cb4-be29-ba73f9135388","Type":"ContainerDied","Data":"13dec8004fee61986a613c5ddab33d216bf61e2c051c4b98a1db8ccd5d33c15f"} Nov 01 22:32:36.037055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.032571 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" Nov 01 22:32:36.037055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.033411 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp" event={"ID":"9f995556-a12a-4021-950a-7130a1467e7e","Type":"ContainerDied","Data":"1cb1b422df1314e557ce6a507104f6196b8d295dd3ff68326dab00485ad6a20e"} Nov 01 22:32:36.037055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.033476 4395 scope.go:122] "RemoveContainer" containerID="38c317f4265cf8edbb3d86a82f1ef79281436ce5bb17fce1f4389fd642f4b2ff" Nov 01 22:32:36.040847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.040808 4395 generic.go:356] "Generic (PLEG): container finished" podID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerID="ec83eab834fc5ee337c32b6e5a96de39c2008224c6ef25b0a8b9553185561202" exitCode=0 Nov 01 22:32:36.041067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.040918 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" event={"ID":"c04819ee-0d0d-4929-b94e-15f87ffc1228","Type":"ContainerDied","Data":"ec83eab834fc5ee337c32b6e5a96de39c2008224c6ef25b0a8b9553185561202"} Nov 01 22:32:36.061924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.061889 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 22:32:36.066204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.066175 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-5qhkp"] Nov 01 22:32:36.132787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.132761 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 22:32:36.219341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.218922 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 22:32:36.220943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.220923 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:36.234419 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.234397 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\") pod \"50de95f8-e75d-4cb4-be29-ba73f9135388\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " Nov 01 22:32:36.234557 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.234457 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\" (UniqueName: \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\") pod \"50de95f8-e75d-4cb4-be29-ba73f9135388\" (UID: \"50de95f8-e75d-4cb4-be29-ba73f9135388\") " Nov 01 22:32:36.234885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.234863 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume" pod "50de95f8-e75d-4cb4-be29-ba73f9135388" (UID: "50de95f8-e75d-4cb4-be29-ba73f9135388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:36.236728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.236698 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz" pod "50de95f8-e75d-4cb4-be29-ba73f9135388" (UID: "50de95f8-e75d-4cb4-be29-ba73f9135388"). InnerVolumeSpecName "kube-api-access-s5bjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.335893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.335860 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\") pod \"c04819ee-0d0d-4929-b94e-15f87ffc1228\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " Nov 01 22:32:36.336108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.335919 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\" (UniqueName: \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\") pod \"c04819ee-0d0d-4929-b94e-15f87ffc1228\" (UID: \"c04819ee-0d0d-4929-b94e-15f87ffc1228\") " Nov 01 22:32:36.336108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.335937 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\" (UniqueName: \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\") pod \"9c931dd0-c825-477e-baf2-a17fef703ead\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " Nov 01 22:32:36.336108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336101 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\") pod \"9c931dd0-c825-477e-baf2-a17fef703ead\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " Nov 01 22:32:36.336319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336162 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\") pod \"9c931dd0-c825-477e-baf2-a17fef703ead\" (UID: \"9c931dd0-c825-477e-baf2-a17fef703ead\") " Nov 01 22:32:36.336441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336406 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume" pod "c04819ee-0d0d-4929-b94e-15f87ffc1228" (UID: "c04819ee-0d0d-4929-b94e-15f87ffc1228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:36.336570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336453 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca" pod "9c931dd0-c825-477e-baf2-a17fef703ead" (UID: "9c931dd0-c825-477e-baf2-a17fef703ead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:36.336641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336571 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.336641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336591 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04819ee-0d0d-4929-b94e-15f87ffc1228-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.336641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336607 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50de95f8-e75d-4cb4-be29-ba73f9135388-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.336641 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.336622 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bjz\" (UniqueName: \"kubernetes.io/projected/50de95f8-e75d-4cb4-be29-ba73f9135388-kube-api-access-s5bjz\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.338325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.338295 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj" pod "9c931dd0-c825-477e-baf2-a17fef703ead" (UID: "9c931dd0-c825-477e-baf2-a17fef703ead"). InnerVolumeSpecName "kube-api-access-59sbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.338832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.338803 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h" pod "c04819ee-0d0d-4929-b94e-15f87ffc1228" (UID: "c04819ee-0d0d-4929-b94e-15f87ffc1228"). InnerVolumeSpecName "kube-api-access-jsk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.338922 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.338872 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca" pod "9c931dd0-c825-477e-baf2-a17fef703ead" (UID: "9c931dd0-c825-477e-baf2-a17fef703ead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:36.381949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.381926 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 22:32:36.436988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.436958 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.437224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437051 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.437224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437147 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.437394 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437366 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.437673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437645 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.437903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437884 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.437996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437917 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.437996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.437963 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.438147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438018 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.438147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438121 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.438253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438197 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\") pod \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\" (UID: \"f02eff61-15a6-42e0-98ed-aeef6ffb8c45\") " Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438787 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsk2h\" (UniqueName: \"kubernetes.io/projected/c04819ee-0d0d-4929-b94e-15f87ffc1228-kube-api-access-jsk2h\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438813 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59sbj\" (UniqueName: \"kubernetes.io/projected/9c931dd0-c825-477e-baf2-a17fef703ead-kube-api-access-59sbj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438829 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438827 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438843 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.438886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.438859 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c931dd0-c825-477e-baf2-a17fef703ead-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.440729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.439100 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.440729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.439638 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.441658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.441348 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "kube-api-access-tg8wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.442071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.442016 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:36.442869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.442848 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:36.443785 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.443758 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo" pod "f02eff61-15a6-42e0-98ed-aeef6ffb8c45" (UID: "f02eff61-15a6-42e0-98ed-aeef6ffb8c45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:32:36.539576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539540 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539572 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539585 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539599 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539611 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8wv\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-kube-api-access-tg8wv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539624 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:36.539779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:36.539637 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02eff61-15a6-42e0-98ed-aeef6ffb8c45-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:37.047985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.047950 4395 generic.go:356] "Generic (PLEG): container finished" podID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerID="51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a" exitCode=0 Nov 01 22:32:37.048514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.048002 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" Nov 01 22:32:37.048514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.048061 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" event={"ID":"f02eff61-15a6-42e0-98ed-aeef6ffb8c45","Type":"ContainerDied","Data":"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a"} Nov 01 22:32:37.048514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.048135 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp" event={"ID":"f02eff61-15a6-42e0-98ed-aeef6ffb8c45","Type":"ContainerDied","Data":"184de348c0c72ca439818e273fb747c5632eabf0d616db95510cdc63be458197"} Nov 01 22:32:37.048514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.048170 4395 scope.go:122] "RemoveContainer" containerID="51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a" Nov 01 22:32:37.049575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.049556 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" Nov 01 22:32:37.049889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.049586 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx" event={"ID":"9c931dd0-c825-477e-baf2-a17fef703ead","Type":"ContainerDied","Data":"ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753"} Nov 01 22:32:37.049889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.049639 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac402f7a4377e9f4020bcd24db52cb739b2996168f36fb89f92934263279b753" Nov 01 22:32:37.054063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.051107 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" Nov 01 22:32:37.054063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.051127 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g" event={"ID":"50de95f8-e75d-4cb4-be29-ba73f9135388","Type":"ContainerDied","Data":"d5f724df42f9b722e89485937546c7f32c5c3553d1e42a562c781e94feb9df91"} Nov 01 22:32:37.056310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.055751 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" event={"ID":"c04819ee-0d0d-4929-b94e-15f87ffc1228","Type":"ContainerDied","Data":"3ac5739fb196b88e5f7d28d61a8495e7d0f10df5767279bbf060f57f4bb79e1c"} Nov 01 22:32:37.056310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.055798 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n" Nov 01 22:32:37.062737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.062709 4395 scope.go:122] "RemoveContainer" containerID="51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a" Nov 01 22:32:37.063056 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:37.063015 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a\": container with ID starting with 51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a not found: ID does not exist" containerID="51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a" Nov 01 22:32:37.063148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.063063 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a"} err="failed to get container status \"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a\": rpc error: code = NotFound desc = could not find container \"51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a\": container with ID starting with 51879c871d5b9730d176683f5864e573df059390cc7da017c7769f7b298ed05a not found: ID does not exist" Nov 01 22:32:37.063148 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.063084 4395 scope.go:122] "RemoveContainer" containerID="13dec8004fee61986a613c5ddab33d216bf61e2c051c4b98a1db8ccd5d33c15f" Nov 01 22:32:37.072794 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.072775 4395 scope.go:122] "RemoveContainer" containerID="ec83eab834fc5ee337c32b6e5a96de39c2008224c6ef25b0a8b9553185561202" Nov 01 22:32:37.082338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.082317 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 22:32:37.084551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.084531 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0def36-55888856c8-zff9g"] Nov 01 22:32:37.098841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.098811 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 22:32:37.101147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.101127 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-rsqjp"] Nov 01 22:32:37.111839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.111819 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 22:32:37.116493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.116475 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0def36-69fb87758c-kcs2n"] Nov 01 22:32:37.176922 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.176891 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" path="/var/lib/kubelet/pods/50de95f8-e75d-4cb4-be29-ba73f9135388/volumes" Nov 01 22:32:37.177452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.177431 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f995556-a12a-4021-950a-7130a1467e7e" path="/var/lib/kubelet/pods/9f995556-a12a-4021-950a-7130a1467e7e/volumes" Nov 01 22:32:37.177836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.177817 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" path="/var/lib/kubelet/pods/c04819ee-0d0d-4929-b94e-15f87ffc1228/volumes" Nov 01 22:32:37.178579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:37.178558 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" path="/var/lib/kubelet/pods/f02eff61-15a6-42e0-98ed-aeef6ffb8c45/volumes" Nov 01 22:32:38.055542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:38.055508 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 22:32:38.066986 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:38.064441 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 22:32:43.247627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.247587 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 22:32:43.716861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.716825 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt"] Nov 01 22:32:43.717551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717526 4395 cpu_manager.go:414] "containerMap: removing container" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" Nov 01 22:32:43.717551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717549 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717566 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9c931dd0-c825-477e-baf2-a17fef703ead" containerName="revoke-keys" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717573 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c931dd0-c825-477e-baf2-a17fef703ead" containerName="revoke-keys" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717592 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717602 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717612 4395 cpu_manager.go:414] "containerMap: removing container" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717620 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717637 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" Nov 01 22:32:43.717781 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717644 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" Nov 01 22:32:43.718253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717787 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="50de95f8-e75d-4cb4-be29-ba73f9135388" containerName="payload-processing" Nov 01 22:32:43.718253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717800 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c931dd0-c825-477e-baf2-a17fef703ead" containerName="revoke-keys" Nov 01 22:32:43.718253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717811 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04819ee-0d0d-4929-b94e-15f87ffc1228" containerName="payload-pre-processing" Nov 01 22:32:43.718253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717820 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f995556-a12a-4021-950a-7130a1467e7e" containerName="istio-proxy" Nov 01 22:32:43.718253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.717829 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02eff61-15a6-42e0-98ed-aeef6ffb8c45" containerName="istio-proxy" Nov 01 22:32:43.721295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.721275 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.731633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.730672 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt"] Nov 01 22:32:43.731633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.731396 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:32:43.731633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.731412 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:43.811997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.811964 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.812187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.812151 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jcs\" (UniqueName: \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.812234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.812205 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.912995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.912961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jcs\" (UniqueName: \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.913176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.913016 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.913176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.913105 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.913810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.913778 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.915841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.915814 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:43.925090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:43.925007 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jcs\" (UniqueName: \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\") pod \"maas-api-revoke-sub-75fcb7c3-h8hkt\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:44.039426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:44.039392 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:44.322279 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:44.322250 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt"] Nov 01 22:32:44.323419 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:32:44.323392 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4470a242_97cb_4f6d_914b_bc18924953f9.slice/crio-62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720 WatchSource:0}: Error finding container 62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720: Status 404 returned error can't find the container with id 62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720 Nov 01 22:32:45.120252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:45.120222 4395 generic.go:356] "Generic (PLEG): container finished" podID="4470a242-97cb-4f6d-914b-bc18924953f9" containerID="199c89f753d97f7f9fe88bea3a33e68339ddace0d18c2e983842551e15d1be48" exitCode=0 Nov 01 22:32:45.120368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:45.120303 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" event={"ID":"4470a242-97cb-4f6d-914b-bc18924953f9","Type":"ContainerDied","Data":"199c89f753d97f7f9fe88bea3a33e68339ddace0d18c2e983842551e15d1be48"} Nov 01 22:32:45.120423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:45.120387 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" event={"ID":"4470a242-97cb-4f6d-914b-bc18924953f9","Type":"ContainerStarted","Data":"62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720"} Nov 01 22:32:46.272210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.272185 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:46.335087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.335063 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\") pod \"4470a242-97cb-4f6d-914b-bc18924953f9\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " Nov 01 22:32:46.335216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.335169 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\" (UniqueName: \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\") pod \"4470a242-97cb-4f6d-914b-bc18924953f9\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " Nov 01 22:32:46.335294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.335222 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\") pod \"4470a242-97cb-4f6d-914b-bc18924953f9\" (UID: \"4470a242-97cb-4f6d-914b-bc18924953f9\") " Nov 01 22:32:46.335416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.335394 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca" pod "4470a242-97cb-4f6d-914b-bc18924953f9" (UID: "4470a242-97cb-4f6d-914b-bc18924953f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:46.337328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.337306 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca" pod "4470a242-97cb-4f6d-914b-bc18924953f9" (UID: "4470a242-97cb-4f6d-914b-bc18924953f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:46.337425 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.337372 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs" pod "4470a242-97cb-4f6d-914b-bc18924953f9" (UID: "4470a242-97cb-4f6d-914b-bc18924953f9"). InnerVolumeSpecName "kube-api-access-62jcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:46.436518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.436456 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:46.436518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.436483 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62jcs\" (UniqueName: \"kubernetes.io/projected/4470a242-97cb-4f6d-914b-bc18924953f9-kube-api-access-62jcs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:46.436518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:46.436498 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4470a242-97cb-4f6d-914b-bc18924953f9-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:47.133078 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:47.133021 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" Nov 01 22:32:47.133248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:47.133048 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt" event={"ID":"4470a242-97cb-4f6d-914b-bc18924953f9","Type":"ContainerDied","Data":"62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720"} Nov 01 22:32:47.133248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:47.133175 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62985e12c2580d0c913507f618b47ff9723460e08a6b1e12760287e26e690720" Nov 01 22:32:48.248825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:48.248778 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" containerID="cri-o://b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856" gracePeriod=25 Nov 01 22:32:52.301190 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:52.301155 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:32:53.082368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.082327 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" containerID="cri-o://935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8" gracePeriod=45 Nov 01 22:32:53.094365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.094339 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" containerID="cri-o://3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f" gracePeriod=45 Nov 01 22:32:53.340566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.340514 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:32:53.369610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.369583 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:32:53.493817 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493787 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493836 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493877 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\" (UniqueName: \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493908 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493927 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493959 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\" (UniqueName: \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.493989 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494060 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494076 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494101 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494149 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\") pod \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\" (UID: \"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494198 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494208 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494225 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\") pod \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\" (UID: \"2aaee67e-3a32-4532-ae18-6adf1da6a58b\") " Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494301 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.494399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494312 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494461 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494664 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494762 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494785 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494799 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494813 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-model-cache\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.495111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.494826 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-home\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.496501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.496475 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "kube-api-access-rwdp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:53.496738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.496708 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "kube-api-access-56rw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:53.496738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.496722 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:53.496738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.496731 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm" pod "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" (UID: "6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.496941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.496904 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:32:53.497066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.497049 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs" pod "2aaee67e-3a32-4532-ae18-6adf1da6a58b" (UID: "2aaee67e-3a32-4532-ae18-6adf1da6a58b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595910 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tmp-dir\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595932 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595941 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdp4\" (UniqueName: \"kubernetes.io/projected/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-kube-api-access-rwdp4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595950 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rw7\" (UniqueName: \"kubernetes.io/projected/2aaee67e-3a32-4532-ae18-6adf1da6a58b-kube-api-access-56rw7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595959 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaee67e-3a32-4532-ae18-6adf1da6a58b-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.595968 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595968 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e-dshm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:53.596273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:53.595975 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaee67e-3a32-4532-ae18-6adf1da6a58b-tls-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:54.176348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.176312 4395 generic.go:356] "Generic (PLEG): container finished" podID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerID="3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f" exitCode=0 Nov 01 22:32:54.176519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.176356 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" Nov 01 22:32:54.176519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.176404 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" event={"ID":"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e","Type":"ContainerDied","Data":"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f"} Nov 01 22:32:54.176519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.176440 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" event={"ID":"6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e","Type":"ContainerDied","Data":"4e7c8852a474356181c5f2082536e37107ac24e4c8c2b7f70057ee398b54eb7e"} Nov 01 22:32:54.176519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.176462 4395 scope.go:122] "RemoveContainer" containerID="3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f" Nov 01 22:32:54.178234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.178211 4395 generic.go:356] "Generic (PLEG): container finished" podID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerID="935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8" exitCode=0 Nov 01 22:32:54.178366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.178244 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" event={"ID":"2aaee67e-3a32-4532-ae18-6adf1da6a58b","Type":"ContainerDied","Data":"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8"} Nov 01 22:32:54.178366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.178282 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" event={"ID":"2aaee67e-3a32-4532-ae18-6adf1da6a58b","Type":"ContainerDied","Data":"4e1e421ffe17740832906a574e8f9d5f6e0656a8eebfe1e4c3e7af80d0f2af08"} Nov 01 22:32:54.178366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.178249 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" Nov 01 22:32:54.188095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.188009 4395 scope.go:122] "RemoveContainer" containerID="3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f" Nov 01 22:32:54.188630 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:54.188324 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f\": container with ID starting with 3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f not found: ID does not exist" containerID="3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f" Nov 01 22:32:54.188630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.188357 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f"} err="failed to get container status \"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f\": rpc error: code = NotFound desc = could not find container \"3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f\": container with ID starting with 3eeaff16bf221fc3c070273a5835ee868aa9f8c11ccf8962661de7ca309c034f not found: ID does not exist" Nov 01 22:32:54.188630 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.188380 4395 scope.go:122] "RemoveContainer" containerID="935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8" Nov 01 22:32:54.199098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.199079 4395 scope.go:122] "RemoveContainer" containerID="935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8" Nov 01 22:32:54.199397 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:32:54.199369 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8\": container with ID starting with 935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8 not found: ID does not exist" containerID="935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8" Nov 01 22:32:54.199451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.199408 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8"} err="failed to get container status \"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8\": rpc error: code = NotFound desc = could not find container \"935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8\": container with ID starting with 935b70555bfe7a828e25ab33ca96ae511bf4d91317a192d8991d1c59c954abc8 not found: ID does not exist" Nov 01 22:32:54.205455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.205432 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 22:32:54.209343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.209324 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj"] Nov 01 22:32:54.221192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.221167 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 22:32:54.225847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.225828 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv"] Nov 01 22:32:54.309273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.309237 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/distinct-w0-1f3a6b-kserve-97c8cd57d-p8qqv" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:32:54.363202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.363111 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f3a6b/premium-w0-1f3a6b-kserve-59cdb44bf9-rzflj" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:32:54.683944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:54.683865 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:32:55.163357 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:55.163331 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" path="/var/lib/kubelet/pods/2aaee67e-3a32-4532-ae18-6adf1da6a58b/volumes" Nov 01 22:32:55.163704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:55.163692 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" path="/var/lib/kubelet/pods/6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e/volumes" Nov 01 22:32:58.970604 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:58.970569 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx"] Nov 01 22:32:58.976089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:58.976051 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f3a6b-xnhqx"] Nov 01 22:32:59.164605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:32:59.164574 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c931dd0-c825-477e-baf2-a17fef703ead" path="/var/lib/kubelet/pods/9c931dd0-c825-477e-baf2-a17fef703ead/volumes" Nov 01 22:33:04.683823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:04.683786 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:33:06.137924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:06.137886 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 22:33:06.138443 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:06.138243 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" containerID="cri-o://a6d531ef6500114cdd33149a1d155d694176cbb8be27485865088958ee5544f5" gracePeriod=30 Nov 01 22:33:08.505649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.505597 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 22:33:08.633675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.633637 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\" (UniqueName: \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\") pod \"272d608c-9bc8-4c68-a86a-5ef2ef123625\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " Nov 01 22:33:08.633844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.633755 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\") pod \"272d608c-9bc8-4c68-a86a-5ef2ef123625\" (UID: \"272d608c-9bc8-4c68-a86a-5ef2ef123625\") " Nov 01 22:33:08.634180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.634155 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume" pod "272d608c-9bc8-4c68-a86a-5ef2ef123625" (UID: "272d608c-9bc8-4c68-a86a-5ef2ef123625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:08.635895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.635865 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj" pod "272d608c-9bc8-4c68-a86a-5ef2ef123625" (UID: "272d608c-9bc8-4c68-a86a-5ef2ef123625"). InnerVolumeSpecName "kube-api-access-w2pjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:08.641989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.641967 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 22:33:08.642560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.642514 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56cb88789f-6nn87" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" containerID="cri-o://14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c" gracePeriod=30 Nov 01 22:33:08.734406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.734385 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272d608c-9bc8-4c68-a86a-5ef2ef123625-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:08.734406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.734407 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pjj\" (UniqueName: \"kubernetes.io/projected/272d608c-9bc8-4c68-a86a-5ef2ef123625-kube-api-access-w2pjj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:08.881563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:08.881536 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 22:33:09.036474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.036396 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\") pod \"6df3d25d-99a3-4171-8ba5-42594ab44775\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " Nov 01 22:33:09.036626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.036484 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\" (UniqueName: \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\") pod \"6df3d25d-99a3-4171-8ba5-42594ab44775\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " Nov 01 22:33:09.036626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.036555 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\") pod \"6df3d25d-99a3-4171-8ba5-42594ab44775\" (UID: \"6df3d25d-99a3-4171-8ba5-42594ab44775\") " Nov 01 22:33:09.038532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.038504 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert" pod "6df3d25d-99a3-4171-8ba5-42594ab44775" (UID: "6df3d25d-99a3-4171-8ba5-42594ab44775"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:09.038661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.038543 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2" pod "6df3d25d-99a3-4171-8ba5-42594ab44775" (UID: "6df3d25d-99a3-4171-8ba5-42594ab44775"). InnerVolumeSpecName "kube-api-access-c4nn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:09.038661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.038598 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs" pod "6df3d25d-99a3-4171-8ba5-42594ab44775" (UID: "6df3d25d-99a3-4171-8ba5-42594ab44775"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:09.138001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.137973 4395 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-webhook-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:09.138001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.137996 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nn2\" (UniqueName: \"kubernetes.io/projected/6df3d25d-99a3-4171-8ba5-42594ab44775-kube-api-access-c4nn2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:09.138180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.138007 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df3d25d-99a3-4171-8ba5-42594ab44775-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:09.270092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.270065 4395 generic.go:356] "Generic (PLEG): container finished" podID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerID="14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c" exitCode=0 Nov 01 22:33:09.270221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.270119 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-6nn87" Nov 01 22:33:09.270221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.270144 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerDied","Data":"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c"} Nov 01 22:33:09.270221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.270192 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-6nn87" event={"ID":"6df3d25d-99a3-4171-8ba5-42594ab44775","Type":"ContainerDied","Data":"943014618cac1a4263762b9cbf8e1f894b25e490847b027f3c6b434855eea9ea"} Nov 01 22:33:09.270221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.270215 4395 scope.go:122] "RemoveContainer" containerID="14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c" Nov 01 22:33:09.271527 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.271501 4395 generic.go:356] "Generic (PLEG): container finished" podID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerID="b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856" exitCode=0 Nov 01 22:33:09.271650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.271540 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" Nov 01 22:33:09.271650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.271589 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" event={"ID":"272d608c-9bc8-4c68-a86a-5ef2ef123625","Type":"ContainerDied","Data":"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856"} Nov 01 22:33:09.271650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.271635 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g" event={"ID":"272d608c-9bc8-4c68-a86a-5ef2ef123625","Type":"ContainerDied","Data":"2d6ea11bd96db22494f3c65918688be816b7ffef258decda8e6f722195f4d816"} Nov 01 22:33:09.281051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.281014 4395 scope.go:122] "RemoveContainer" containerID="ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5" Nov 01 22:33:09.288430 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.288377 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 22:33:09.289390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.289376 4395 scope.go:122] "RemoveContainer" containerID="14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c" Nov 01 22:33:09.289702 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:09.289677 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c\": container with ID starting with 14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c not found: ID does not exist" containerID="14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c" Nov 01 22:33:09.289805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.289706 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c"} err="failed to get container status \"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c\": rpc error: code = NotFound desc = could not find container \"14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c\": container with ID starting with 14dbe3623c614ea6c6adf7f02734a853a5b9705518bd465340ffbd61790be84c not found: ID does not exist" Nov 01 22:33:09.289805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.289723 4395 scope.go:122] "RemoveContainer" containerID="ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5" Nov 01 22:33:09.290004 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:09.289973 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5\": container with ID starting with ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5 not found: ID does not exist" containerID="ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5" Nov 01 22:33:09.290128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.290018 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5"} err="failed to get container status \"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5\": rpc error: code = NotFound desc = could not find container \"ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5\": container with ID starting with ce71f02746aea4b45ba6b6751e997f8670107e22a8d79159eb48cce5cde197f5 not found: ID does not exist" Nov 01 22:33:09.290128 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.290056 4395 scope.go:122] "RemoveContainer" containerID="b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856" Nov 01 22:33:09.292862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.292838 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f3a6b-686494f97b-8st9g"] Nov 01 22:33:09.298317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.298300 4395 scope.go:122] "RemoveContainer" containerID="b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856" Nov 01 22:33:09.298557 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:09.298539 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856\": container with ID starting with b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856 not found: ID does not exist" containerID="b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856" Nov 01 22:33:09.298628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.298568 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856"} err="failed to get container status \"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856\": rpc error: code = NotFound desc = could not find container \"b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856\": container with ID starting with b74ea9deef7a97c8d29e756033efdad305718a9d88ac8c5e8ba4960a2c662856 not found: ID does not exist" Nov 01 22:33:09.303711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.303691 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 22:33:09.307598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.307579 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56cb88789f-6nn87"] Nov 01 22:33:09.957568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.957537 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.957975 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4470a242-97cb-4f6d-914b-bc18924953f9" containerName="revoke-keys" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.957992 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4470a242-97cb-4f6d-914b-bc18924953f9" containerName="revoke-keys" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958017 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958040 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958063 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958072 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958096 4395 cpu_manager.go:414] "containerMap: removing container" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958102 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958112 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.958115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958117 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958124 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958132 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958221 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" containerName="payload-processing" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958233 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958242 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff4e66d-2207-4b0d-bf2e-e9a5dc62b41e" containerName="main" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958252 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4470a242-97cb-4f6d-914b-bc18924953f9" containerName="revoke-keys" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958264 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaee67e-3a32-4532-ae18-6adf1da6a58b" containerName="main" Nov 01 22:33:09.958608 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.958273 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" containerName="manager" Nov 01 22:33:09.963062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.963044 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:09.971671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.971648 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ntfs9\"" Nov 01 22:33:09.971815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.971694 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 22:33:09.971815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.971775 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 22:33:09.973478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.973450 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:09.991556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:09.991536 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 22:33:10.147846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.147816 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.148014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.147862 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.148014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.147963 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mmj\" (UniqueName: \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.248774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.248692 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.248774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.248738 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.248774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.248765 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mmj\" (UniqueName: \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.251293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.251264 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.251432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.251307 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.261236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.261209 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mmj\" (UniqueName: \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\") pod \"maas-controller-56cb88789f-g7264\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.281194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.281166 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:10.564198 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:10.564175 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:10.566351 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:33:10.566319 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351741b4_413d_4345_82cc_48826f76112e.slice/crio-57a0cfb35e17bccf905f057b970ebd692883305f65012e75b07fd13489bdf08b WatchSource:0}: Error finding container 57a0cfb35e17bccf905f057b970ebd692883305f65012e75b07fd13489bdf08b: Status 404 returned error can't find the container with id 57a0cfb35e17bccf905f057b970ebd692883305f65012e75b07fd13489bdf08b Nov 01 22:33:11.163823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.163797 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272d608c-9bc8-4c68-a86a-5ef2ef123625" path="/var/lib/kubelet/pods/272d608c-9bc8-4c68-a86a-5ef2ef123625/volumes" Nov 01 22:33:11.164246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.164230 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df3d25d-99a3-4171-8ba5-42594ab44775" path="/var/lib/kubelet/pods/6df3d25d-99a3-4171-8ba5-42594ab44775/volumes" Nov 01 22:33:11.290528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.290488 4395 generic.go:356] "Generic (PLEG): container finished" podID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerID="a6d531ef6500114cdd33149a1d155d694176cbb8be27485865088958ee5544f5" exitCode=0 Nov 01 22:33:11.290744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.290644 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" event={"ID":"8ea09ab9-9560-4d24-9af0-f7520a375323","Type":"ContainerDied","Data":"a6d531ef6500114cdd33149a1d155d694176cbb8be27485865088958ee5544f5"} Nov 01 22:33:11.292892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.292854 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-g7264" event={"ID":"351741b4-413d-4345-82cc-48826f76112e","Type":"ContainerStarted","Data":"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5"} Nov 01 22:33:11.293060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.292915 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-g7264" event={"ID":"351741b4-413d-4345-82cc-48826f76112e","Type":"ContainerStarted","Data":"57a0cfb35e17bccf905f057b970ebd692883305f65012e75b07fd13489bdf08b"} Nov 01 22:33:11.293466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.293443 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:11.319995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.319939 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56cb88789f-g7264" podStartSLOduration=1.914567366 podStartE2EDuration="2.319920041s" totalImagesPullingTime="405.352675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:11.318126318 +0000 UTC m=+2938.292028859" watchObservedRunningTime="2026-11-01 22:33:11.319920041 +0000 UTC m=+2938.293822581" Nov 01 22:33:11.409918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.409890 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 22:33:11.562134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562057 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562091 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562126 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562159 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562181 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562215 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562296 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562321 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562329 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.562410 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\") pod \"8ea09ab9-9560-4d24-9af0-f7520a375323\" (UID: \"8ea09ab9-9560-4d24-9af0-f7520a375323\") " Nov 01 22:33:11.562808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562583 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.562808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562663 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.562808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562773 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.563195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562978 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:11.563195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.562835 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.564433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.564408 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:33:11.564650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.564621 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:11.564650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.564635 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:11.564650 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.564648 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t" pod "8ea09ab9-9560-4d24-9af0-f7520a375323" (UID: "8ea09ab9-9560-4d24-9af0-f7520a375323"). InnerVolumeSpecName "kube-api-access-6x68t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:11.663590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663552 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663583 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663594 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663606 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x68t\" (UniqueName: \"kubernetes.io/projected/8ea09ab9-9560-4d24-9af0-f7520a375323-kube-api-access-6x68t\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663615 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663625 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663633 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea09ab9-9560-4d24-9af0-f7520a375323-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:11.663846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:11.663641 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea09ab9-9560-4d24-9af0-f7520a375323-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:12.300294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:12.300257 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" event={"ID":"8ea09ab9-9560-4d24-9af0-f7520a375323","Type":"ContainerDied","Data":"32847cd1f9b38050787e05f494219382bae0c7179d6cfe3e1e573489f7fd54e7"} Nov 01 22:33:12.300756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:12.300311 4395 scope.go:122] "RemoveContainer" containerID="a6d531ef6500114cdd33149a1d155d694176cbb8be27485865088958ee5544f5" Nov 01 22:33:12.300756 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:12.300347 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7" Nov 01 22:33:12.327704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:12.327678 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 22:33:12.331549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:12.331524 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-pcrl7"] Nov 01 22:33:13.162936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:13.162908 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" path="/var/lib/kubelet/pods/8ea09ab9-9560-4d24-9af0-f7520a375323/volumes" Nov 01 22:33:22.300205 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:22.300159 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:33:22.303376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:22.303354 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:29.836102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.836058 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9"] Nov 01 22:33:29.836660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.836507 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" Nov 01 22:33:29.836660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.836520 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" Nov 01 22:33:29.836660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.836598 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea09ab9-9560-4d24-9af0-f7520a375323" containerName="istio-proxy" Nov 01 22:33:29.841548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.841528 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:29.846459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.846429 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:33:29.846619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.846607 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:33:29.855125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.855099 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9"] Nov 01 22:33:29.919493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.919456 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:29.919645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.919502 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288w8\" (UniqueName: \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:29.919645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:29.919591 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.020405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.020371 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.020569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.020433 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.020569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.020458 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288w8\" (UniqueName: \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.021057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.021015 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.022887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.022868 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.032473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.032453 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288w8\" (UniqueName: \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\") pod \"maas-api-revoke-sub-85237e45-lkmf9\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.159207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.159142 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:30.437252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:30.437014 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9"] Nov 01 22:33:30.440072 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:33:30.440018 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f27c63b_520a_46b5_8680_c08721cbbe01.slice/crio-3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b WatchSource:0}: Error finding container 3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b: Status 404 returned error can't find the container with id 3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b Nov 01 22:33:31.405447 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:31.405354 4395 generic.go:356] "Generic (PLEG): container finished" podID="5f27c63b-520a-46b5-8680-c08721cbbe01" containerID="e278ed64a3dd3abd5d101b1d12f8f536954ca2c9575a028a8cc49ffe81a5e6f5" exitCode=0 Nov 01 22:33:31.405870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:31.405446 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" event={"ID":"5f27c63b-520a-46b5-8680-c08721cbbe01","Type":"ContainerDied","Data":"e278ed64a3dd3abd5d101b1d12f8f536954ca2c9575a028a8cc49ffe81a5e6f5"} Nov 01 22:33:31.405870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:31.405529 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" event={"ID":"5f27c63b-520a-46b5-8680-c08721cbbe01","Type":"ContainerStarted","Data":"3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b"} Nov 01 22:33:32.558437 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.558414 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:32.642324 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.642299 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\") pod \"5f27c63b-520a-46b5-8680-c08721cbbe01\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " Nov 01 22:33:32.642487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.642387 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\" (UniqueName: \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\") pod \"5f27c63b-520a-46b5-8680-c08721cbbe01\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " Nov 01 22:33:32.642487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.642408 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\") pod \"5f27c63b-520a-46b5-8680-c08721cbbe01\" (UID: \"5f27c63b-520a-46b5-8680-c08721cbbe01\") " Nov 01 22:33:32.642835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.642810 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca" pod "5f27c63b-520a-46b5-8680-c08721cbbe01" (UID: "5f27c63b-520a-46b5-8680-c08721cbbe01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:32.644482 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.644451 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca" pod "5f27c63b-520a-46b5-8680-c08721cbbe01" (UID: "5f27c63b-520a-46b5-8680-c08721cbbe01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:32.644567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.644507 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8" pod "5f27c63b-520a-46b5-8680-c08721cbbe01" (UID: "5f27c63b-520a-46b5-8680-c08721cbbe01"). InnerVolumeSpecName "kube-api-access-288w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:32.743687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.743628 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288w8\" (UniqueName: \"kubernetes.io/projected/5f27c63b-520a-46b5-8680-c08721cbbe01-kube-api-access-288w8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:32.743687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.743654 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:32.743687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:32.743665 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f27c63b-520a-46b5-8680-c08721cbbe01-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:33.416894 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:33.416868 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" Nov 01 22:33:33.417065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:33.416910 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9" event={"ID":"5f27c63b-520a-46b5-8680-c08721cbbe01","Type":"ContainerDied","Data":"3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b"} Nov 01 22:33:33.417065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:33.416959 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee3c6bad6dd5aca6e7b2de8527b874c4b10a19fd515eb9e45c6f0d0160a696b" Nov 01 22:33:35.032147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:35.032109 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n"] Nov 01 22:33:35.035379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:35.035355 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6801004e-gvc9n"] Nov 01 22:33:35.163005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:35.162976 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20caf657-54b6-4ba0-a793-8284b6e3ec65" path="/var/lib/kubelet/pods/20caf657-54b6-4ba0-a793-8284b6e3ec65/volumes" Nov 01 22:33:38.033099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.033059 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:33:38.033504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.033253 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerName="discovery" containerID="cri-o://40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef" gracePeriod=30 Nov 01 22:33:38.285806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.285755 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:33:38.394687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394660 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.394824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394702 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.394824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394760 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.394824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394795 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.395077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394832 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.395077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394911 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.395077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.394945 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\") pod \"c8c936de-eaa8-4ca4-af0c-6634d535841a\" (UID: \"c8c936de-eaa8-4ca4-af0c-6634d535841a\") " Nov 01 22:33:38.395309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.395282 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:38.397223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397186 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:38.397344 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397314 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "kube-api-access-vwn4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:38.397649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397620 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:38.397768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397660 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:33:38.397768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397668 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:38.397903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.397873 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert" pod "c8c936de-eaa8-4ca4-af0c-6634d535841a" (UID: "c8c936de-eaa8-4ca4-af0c-6634d535841a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:38.445561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.445526 4395 generic.go:356] "Generic (PLEG): container finished" podID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerID="40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef" exitCode=0 Nov 01 22:33:38.445689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.445584 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" Nov 01 22:33:38.445689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.445613 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" event={"ID":"c8c936de-eaa8-4ca4-af0c-6634d535841a","Type":"ContainerDied","Data":"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef"} Nov 01 22:33:38.445689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.445664 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7" event={"ID":"c8c936de-eaa8-4ca4-af0c-6634d535841a","Type":"ContainerDied","Data":"cb1d5d50336a21b93434c80a1814266aa212cc6e2a54a1a57fc3575be639bd63"} Nov 01 22:33:38.445689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.445688 4395 scope.go:122] "RemoveContainer" containerID="40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef" Nov 01 22:33:38.455496 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.455482 4395 scope.go:122] "RemoveContainer" containerID="40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef" Nov 01 22:33:38.455773 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:38.455755 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef\": container with ID starting with 40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef not found: ID does not exist" containerID="40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef" Nov 01 22:33:38.455812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.455782 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef"} err="failed to get container status \"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef\": rpc error: code = NotFound desc = could not find container \"40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef\": container with ID starting with 40b1c10d8d7015a9265fab48d79c3b4b939d75c12ae6d76ec83ed4eed420beef not found: ID does not exist" Nov 01 22:33:38.468697 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.468674 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:33:38.472926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.472905 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dcjl7"] Nov 01 22:33:38.496066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496044 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwn4w\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-kube-api-access-vwn4w\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496065 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-ca-configmap\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496078 4395 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-cacerts\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496087 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496096 4395 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c936de-eaa8-4ca4-af0c-6634d535841a-local-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496104 4395 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-kubeconfig\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:38.496192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:38.496112 4395 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8c936de-eaa8-4ca4-af0c-6634d535841a-istio-csr-dns-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:39.162416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:39.162385 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" path="/var/lib/kubelet/pods/c8c936de-eaa8-4ca4-af0c-6634d535841a/volumes" Nov 01 22:33:41.381149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.381114 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:41.381560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.381307 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56cb88789f-g7264" podUID="351741b4-413d-4345-82cc-48826f76112e" containerName="manager" containerID="cri-o://fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5" gracePeriod=30 Nov 01 22:33:41.627760 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.627737 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:41.725068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.724969 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\") pod \"351741b4-413d-4345-82cc-48826f76112e\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " Nov 01 22:33:41.725236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.725068 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\") pod \"351741b4-413d-4345-82cc-48826f76112e\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " Nov 01 22:33:41.725236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.725133 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\" (UniqueName: \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\") pod \"351741b4-413d-4345-82cc-48826f76112e\" (UID: \"351741b4-413d-4345-82cc-48826f76112e\") " Nov 01 22:33:41.727175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.727144 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert" pod "351741b4-413d-4345-82cc-48826f76112e" (UID: "351741b4-413d-4345-82cc-48826f76112e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:41.727303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.727263 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs" pod "351741b4-413d-4345-82cc-48826f76112e" (UID: "351741b4-413d-4345-82cc-48826f76112e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:41.727349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.727330 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj" pod "351741b4-413d-4345-82cc-48826f76112e" (UID: "351741b4-413d-4345-82cc-48826f76112e"). InnerVolumeSpecName "kube-api-access-44mmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:41.825862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.825833 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.825862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.825856 4395 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/351741b4-413d-4345-82cc-48826f76112e-webhook-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.826012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:41.825868 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mmj\" (UniqueName: \"kubernetes.io/projected/351741b4-413d-4345-82cc-48826f76112e-kube-api-access-44mmj\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:42.468233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.468200 4395 generic.go:356] "Generic (PLEG): container finished" podID="351741b4-413d-4345-82cc-48826f76112e" containerID="fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5" exitCode=0 Nov 01 22:33:42.468681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.468254 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-g7264" Nov 01 22:33:42.468681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.468278 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-g7264" event={"ID":"351741b4-413d-4345-82cc-48826f76112e","Type":"ContainerDied","Data":"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5"} Nov 01 22:33:42.468681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.468346 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-g7264" event={"ID":"351741b4-413d-4345-82cc-48826f76112e","Type":"ContainerDied","Data":"57a0cfb35e17bccf905f057b970ebd692883305f65012e75b07fd13489bdf08b"} Nov 01 22:33:42.468681 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.468375 4395 scope.go:122] "RemoveContainer" containerID="fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5" Nov 01 22:33:42.478475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.478460 4395 scope.go:122] "RemoveContainer" containerID="fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5" Nov 01 22:33:42.478771 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:42.478746 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5\": container with ID starting with fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5 not found: ID does not exist" containerID="fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5" Nov 01 22:33:42.478874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.478778 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5"} err="failed to get container status \"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5\": rpc error: code = NotFound desc = could not find container \"fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5\": container with ID starting with fc8608b0e05667d9cc3ae392e9896f54cf8f62f200d91c9701a7d9c1e3285ba5 not found: ID does not exist" Nov 01 22:33:42.491474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.491451 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:42.495397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:42.495376 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56cb88789f-g7264"] Nov 01 22:33:43.162741 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.162702 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351741b4-413d-4345-82cc-48826f76112e" path="/var/lib/kubelet/pods/351741b4-413d-4345-82cc-48826f76112e/volumes" Nov 01 22:33:43.655412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655387 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56cb88789f-pxh6p"] Nov 01 22:33:43.655845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655826 4395 cpu_manager.go:414] "containerMap: removing container" podUID="351741b4-413d-4345-82cc-48826f76112e" containerName="manager" Nov 01 22:33:43.655934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655848 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351741b4-413d-4345-82cc-48826f76112e" containerName="manager" Nov 01 22:33:43.655934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655869 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5f27c63b-520a-46b5-8680-c08721cbbe01" containerName="revoke-keys" Nov 01 22:33:43.655934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655878 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f27c63b-520a-46b5-8680-c08721cbbe01" containerName="revoke-keys" Nov 01 22:33:43.655934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655920 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerName="discovery" Nov 01 22:33:43.655934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.655929 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerName="discovery" Nov 01 22:33:43.656159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.656049 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c936de-eaa8-4ca4-af0c-6634d535841a" containerName="discovery" Nov 01 22:33:43.656159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.656071 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="351741b4-413d-4345-82cc-48826f76112e" containerName="manager" Nov 01 22:33:43.656159 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.656083 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f27c63b-520a-46b5-8680-c08721cbbe01" containerName="revoke-keys" Nov 01 22:33:43.659567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.659552 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.664350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.664330 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 22:33:43.664470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.664380 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ntfs9\"" Nov 01 22:33:43.664470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.664429 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 22:33:43.664570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.664481 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 22:33:43.670407 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.670384 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-pxh6p"] Nov 01 22:33:43.742249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.742224 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-metrics-certs\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.742371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.742257 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9ph\" (UniqueName: \"kubernetes.io/projected/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-kube-api-access-hv9ph\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.742371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.742282 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-webhook-cert\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.843521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.843492 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-metrics-certs\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.843687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.843528 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9ph\" (UniqueName: \"kubernetes.io/projected/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-kube-api-access-hv9ph\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.843687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.843553 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-webhook-cert\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.846095 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.846075 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-metrics-certs\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.846228 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.846212 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-webhook-cert\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.855516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.855494 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9ph\" (UniqueName: \"kubernetes.io/projected/99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a-kube-api-access-hv9ph\") pod \"maas-controller-56cb88789f-pxh6p\" (UID: \"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a\") " pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:43.973637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:43.973583 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:44.251914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:44.251891 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56cb88789f-pxh6p"] Nov 01 22:33:44.253817 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:33:44.253789 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ab3f18_0a51_4d73_a8b7_ff9866dbeb2a.slice/crio-d9c58615d1a3e32af11c36c4affbf8d4a121190ea86d6013713856479ad278db WatchSource:0}: Error finding container d9c58615d1a3e32af11c36c4affbf8d4a121190ea86d6013713856479ad278db: Status 404 returned error can't find the container with id d9c58615d1a3e32af11c36c4affbf8d4a121190ea86d6013713856479ad278db Nov 01 22:33:44.255102 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:44.255081 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:33:44.482305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:44.482267 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-pxh6p" event={"ID":"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a","Type":"ContainerStarted","Data":"d9c58615d1a3e32af11c36c4affbf8d4a121190ea86d6013713856479ad278db"} Nov 01 22:33:45.489438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:45.489402 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56cb88789f-pxh6p" event={"ID":"99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a","Type":"ContainerStarted","Data":"e681e13e8ebd360da55040e6b454267070ae553d11e81007c50021b0a508a16b"} Nov 01 22:33:45.489873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:45.489527 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:33:45.514843 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:45.514798 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56cb88789f-pxh6p" podStartSLOduration=1.914606063 podStartE2EDuration="2.514787044s" totalImagesPullingTime="600.180981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:45.512351907 +0000 UTC m=+2972.486254482" watchObservedRunningTime="2026-11-01 22:33:45.514787044 +0000 UTC m=+2972.488689582" Nov 01 22:33:46.758018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.757980 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g"] Nov 01 22:33:46.761846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.761827 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.776387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.776361 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g"] Nov 01 22:33:46.874107 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874078 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874115 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874140 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874163 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874312 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lj6f\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-kube-api-access-8lj6f\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874362 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874427 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874477 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.874565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.874521 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975284 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975321 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975324 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975359 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975378 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975400 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975421 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975471 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975513 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lj6f\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-kube-api-access-8lj6f\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975850 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.975985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.975861 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.976162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.976137 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.976246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.976175 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.976367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.976343 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.978438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.978417 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.978628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.978606 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.988469 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.988449 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:46.988568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:46.988507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lj6f\" (UniqueName: \"kubernetes.io/projected/cdbb4f72-c168-4cba-ba1a-58b7eebc517f-kube-api-access-8lj6f\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cjw9g\" (UID: \"cdbb4f72-c168-4cba-ba1a-58b7eebc517f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:47.075460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.075399 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:47.359538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.359500 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g"] Nov 01 22:33:47.361765 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:33:47.361742 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbb4f72_c168_4cba_ba1a_58b7eebc517f.slice/crio-c57241830ab963b84e952eab22bca6e26462fb85de90149f1b7a2c0c9f0140e9 WatchSource:0}: Error finding container c57241830ab963b84e952eab22bca6e26462fb85de90149f1b7a2c0c9f0140e9: Status 404 returned error can't find the container with id c57241830ab963b84e952eab22bca6e26462fb85de90149f1b7a2c0c9f0140e9 Nov 01 22:33:47.363829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.363801 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:33:47.363892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.363861 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:33:47.363892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.363886 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:33:47.501626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.501591 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" event={"ID":"cdbb4f72-c168-4cba-ba1a-58b7eebc517f","Type":"ContainerStarted","Data":"d3bb5919fee435a30547230fa5ca15327eddb17cc156bef613a9162b4f1171f1"} Nov 01 22:33:47.501789 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.501652 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" event={"ID":"cdbb4f72-c168-4cba-ba1a-58b7eebc517f","Type":"ContainerStarted","Data":"c57241830ab963b84e952eab22bca6e26462fb85de90149f1b7a2c0c9f0140e9"} Nov 01 22:33:47.526134 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:47.526077 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podStartSLOduration=1.526058572 podStartE2EDuration="1.526058572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:47.523713003 +0000 UTC m=+2974.497615542" watchObservedRunningTime="2026-11-01 22:33:47.526058572 +0000 UTC m=+2974.499961111" Nov 01 22:33:48.097562 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:48.097523 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:48.098381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:48.098355 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:48.098576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:48.098548 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:49.094054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:49.087452 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:49.094054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:49.087514 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:50.095053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:50.092398 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:50.095053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:50.092461 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:51.095054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:51.087455 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:51.095054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:51.087528 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:52.095054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:52.089411 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:52.095054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:52.089477 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:52.324110 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:33:52.324077 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:33:53.091251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:53.091219 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:53.091550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:53.091520 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:54.075983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:54.075946 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.076379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:54.076005 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 22:33:55.475712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:55.475678 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:55.476211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:55.476067 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:55.477057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:55.477016 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" Nov 01 22:33:56.496454 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:33:56.496425 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56cb88789f-pxh6p" Nov 01 22:34:01.776634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.776599 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2"] Nov 01 22:34:01.780531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.780500 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.793331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.793305 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2"] Nov 01 22:34:01.822138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822109 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822145 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822166 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822221 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqw97\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-kube-api-access-bqw97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822259 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822281 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822355 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822394 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.822503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.822424 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923384 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923417 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923445 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqw97\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-kube-api-access-bqw97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923479 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923498 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923529 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923554 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.923910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923639 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.924018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.923991 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.924135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.924022 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.924135 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.924101 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.924227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.924158 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.924227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.924197 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.926086 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.926068 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.926227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.926211 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.935634 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.935612 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqw97\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-kube-api-access-bqw97\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:01.935883 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:01.935866 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ca1b5-2ea4-4aef-8432-b6a745e3bd4a-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-gwqw2\" (UID: \"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:02.094090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.094021 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:02.382367 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.382344 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2"] Nov 01 22:34:02.384082 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:34:02.384054 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071ca1b5_2ea4_4aef_8432_b6a745e3bd4a.slice/crio-89a66dc3b537fe5dd7b36a24d92d2a498fe224956ddd4d58fda0df4ec71b85eb WatchSource:0}: Error finding container 89a66dc3b537fe5dd7b36a24d92d2a498fe224956ddd4d58fda0df4ec71b85eb: Status 404 returned error can't find the container with id 89a66dc3b537fe5dd7b36a24d92d2a498fe224956ddd4d58fda0df4ec71b85eb Nov 01 22:34:02.386200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.386171 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:02.386295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.386231 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:02.386295 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.386256 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:02.587501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.587454 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" event={"ID":"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a","Type":"ContainerStarted","Data":"c25881e1aef52b881b0e5e9ea867cc248bda983f7d8eda7d485aacb20959e0c3"} Nov 01 22:34:02.587719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.587534 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" event={"ID":"071ca1b5-2ea4-4aef-8432-b6a745e3bd4a","Type":"ContainerStarted","Data":"89a66dc3b537fe5dd7b36a24d92d2a498fe224956ddd4d58fda0df4ec71b85eb"} Nov 01 22:34:02.613628 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:02.613574 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podStartSLOduration=1.613558842 podStartE2EDuration="1.613558842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:02.61117821 +0000 UTC m=+2989.585080751" watchObservedRunningTime="2026-11-01 22:34:02.613558842 +0000 UTC m=+2989.587461470" Nov 01 22:34:03.100298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:03.100261 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:03.101873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:03.101837 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:03.102070 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:03.101898 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:04.099474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:04.099428 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:04.099657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:04.099493 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:05.099468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:05.099427 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:05.099956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:05.099496 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:06.097450 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:06.097411 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.097786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:06.097757 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:07.099405 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:07.099368 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.099845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:07.099429 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:08.098053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:08.095393 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.098053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:08.095455 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:09.094502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:09.094471 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 22:34:09.094923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:09.094522 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 22:34:10.546964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:10.546928 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:10.547402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:10.547323 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:10.548265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:10.548243 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" Nov 01 22:34:22.300955 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:34:22.300915 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:34:24.374575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.374489 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm"] Nov 01 22:34:24.383059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.382609 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.384673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.384644 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jlq\" (UniqueName: \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.384821 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.384719 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.384931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.384908 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.386723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.386695 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:24.387850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.387813 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:34:24.388328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.388307 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm"] Nov 01 22:34:24.486147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.486109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jlq\" (UniqueName: \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.486370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.486162 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.486370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.486226 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.487068 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.486958 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.488812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.488784 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.498518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.498497 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jlq\" (UniqueName: \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\") pod \"maas-api-revoke-sub-c1e271a9-sdspm\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.699113 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.699049 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:24.974944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:24.974857 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm"] Nov 01 22:34:24.978119 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:34:24.978090 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74102a24_7d48_4bb4_8e65_53ae0d45b6a1.slice/crio-17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891 WatchSource:0}: Error finding container 17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891: Status 404 returned error can't find the container with id 17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891 Nov 01 22:34:25.770801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:25.770761 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" event={"ID":"74102a24-7d48-4bb4-8e65-53ae0d45b6a1","Type":"ContainerStarted","Data":"1846ea622d0e0882c200a70d6dd4853deb00ed028009b95d623a95f334b7173f"} Nov 01 22:34:25.771302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:25.770822 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" event={"ID":"74102a24-7d48-4bb4-8e65-53ae0d45b6a1","Type":"ContainerStarted","Data":"17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891"} Nov 01 22:34:25.789733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:25.789689 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" podStartSLOduration=1.789674549 podStartE2EDuration="1.789674549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:25.788554533 +0000 UTC m=+3012.762457074" watchObservedRunningTime="2026-11-01 22:34:25.789674549 +0000 UTC m=+3012.763577087" Nov 01 22:34:26.777514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:26.777480 4395 generic.go:356] "Generic (PLEG): container finished" podID="74102a24-7d48-4bb4-8e65-53ae0d45b6a1" containerID="1846ea622d0e0882c200a70d6dd4853deb00ed028009b95d623a95f334b7173f" exitCode=0 Nov 01 22:34:26.777909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:26.777576 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" event={"ID":"74102a24-7d48-4bb4-8e65-53ae0d45b6a1","Type":"ContainerDied","Data":"1846ea622d0e0882c200a70d6dd4853deb00ed028009b95d623a95f334b7173f"} Nov 01 22:34:27.916617 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:27.916593 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:28.013257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.013230 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\") pod \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " Nov 01 22:34:28.013414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.013294 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\" (UniqueName: \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\") pod \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " Nov 01 22:34:28.013414 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.013319 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\") pod \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\" (UID: \"74102a24-7d48-4bb4-8e65-53ae0d45b6a1\") " Nov 01 22:34:28.013672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.013644 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca" pod "74102a24-7d48-4bb4-8e65-53ae0d45b6a1" (UID: "74102a24-7d48-4bb4-8e65-53ae0d45b6a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:28.015561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.015542 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca" pod "74102a24-7d48-4bb4-8e65-53ae0d45b6a1" (UID: "74102a24-7d48-4bb4-8e65-53ae0d45b6a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:28.015635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.015576 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq" pod "74102a24-7d48-4bb4-8e65-53ae0d45b6a1" (UID: "74102a24-7d48-4bb4-8e65-53ae0d45b6a1"). InnerVolumeSpecName "kube-api-access-b9jlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:28.114975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.114915 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:28.114975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.114941 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jlq\" (UniqueName: \"kubernetes.io/projected/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-kube-api-access-b9jlq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:28.114975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.114951 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74102a24-7d48-4bb4-8e65-53ae0d45b6a1-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:28.790152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.790110 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" event={"ID":"74102a24-7d48-4bb4-8e65-53ae0d45b6a1","Type":"ContainerDied","Data":"17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891"} Nov 01 22:34:28.790374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.790157 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm" Nov 01 22:34:28.790374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:28.790167 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17253dd877126f480af0658d080224dd86017d2fc67dace0a0280ff45cd42891" Nov 01 22:34:43.084099 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.084060 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4"] Nov 01 22:34:43.086140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.086110 4395 cpu_manager.go:414] "containerMap: removing container" podUID="74102a24-7d48-4bb4-8e65-53ae0d45b6a1" containerName="revoke-keys" Nov 01 22:34:43.086336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.086317 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74102a24-7d48-4bb4-8e65-53ae0d45b6a1" containerName="revoke-keys" Nov 01 22:34:43.086837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.086811 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="74102a24-7d48-4bb4-8e65-53ae0d45b6a1" containerName="revoke-keys" Nov 01 22:34:43.091565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.091541 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.123459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.123431 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:34:43.123598 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.123469 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:43.125723 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.125700 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4"] Nov 01 22:34:43.181687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.181653 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7qdl\" (UniqueName: \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.181849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.181719 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.181849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.181792 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.282859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.282825 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.283093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.282887 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.283093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.282952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7qdl\" (UniqueName: \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.283555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.283529 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.285432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.285413 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.295624 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.295604 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7qdl\" (UniqueName: \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\") pod \"maas-api-revoke-sub-2e1f5b5f-m9xg4\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.404788 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.404716 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:43.681094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.681070 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4"] Nov 01 22:34:43.682998 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:34:43.682972 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e20e20c_a016_430c_a065_d44acb05e5db.slice/crio-559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8 WatchSource:0}: Error finding container 559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8: Status 404 returned error can't find the container with id 559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8 Nov 01 22:34:43.878331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:43.878296 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" event={"ID":"2e20e20c-a016-430c-a065-d44acb05e5db","Type":"ContainerStarted","Data":"559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8"} Nov 01 22:34:44.885275 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:44.885184 4395 generic.go:356] "Generic (PLEG): container finished" podID="2e20e20c-a016-430c-a065-d44acb05e5db" containerID="73edcfca1acf79f94108a1e18d9a56d95a4b45c75c38a7f3c55b360846901f9f" exitCode=0 Nov 01 22:34:44.885644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:44.885275 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" event={"ID":"2e20e20c-a016-430c-a065-d44acb05e5db","Type":"ContainerDied","Data":"73edcfca1acf79f94108a1e18d9a56d95a4b45c75c38a7f3c55b360846901f9f"} Nov 01 22:34:46.029374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.029347 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:46.105637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.105609 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\") pod \"2e20e20c-a016-430c-a065-d44acb05e5db\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " Nov 01 22:34:46.105791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.105665 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\" (UniqueName: \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\") pod \"2e20e20c-a016-430c-a065-d44acb05e5db\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " Nov 01 22:34:46.105791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.105729 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\") pod \"2e20e20c-a016-430c-a065-d44acb05e5db\" (UID: \"2e20e20c-a016-430c-a065-d44acb05e5db\") " Nov 01 22:34:46.106010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.105989 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca" pod "2e20e20c-a016-430c-a065-d44acb05e5db" (UID: "2e20e20c-a016-430c-a065-d44acb05e5db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:46.106103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.106089 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:46.107898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.107878 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca" pod "2e20e20c-a016-430c-a065-d44acb05e5db" (UID: "2e20e20c-a016-430c-a065-d44acb05e5db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:46.107963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.107936 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl" pod "2e20e20c-a016-430c-a065-d44acb05e5db" (UID: "2e20e20c-a016-430c-a065-d44acb05e5db"). InnerVolumeSpecName "kube-api-access-x7qdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:46.206662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.206599 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e20e20c-a016-430c-a065-d44acb05e5db-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:46.206662 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.206622 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7qdl\" (UniqueName: \"kubernetes.io/projected/2e20e20c-a016-430c-a065-d44acb05e5db-kube-api-access-x7qdl\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:46.848050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.847992 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn"] Nov 01 22:34:46.848490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.848471 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2e20e20c-a016-430c-a065-d44acb05e5db" containerName="revoke-keys" Nov 01 22:34:46.848593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.848493 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e20e20c-a016-430c-a065-d44acb05e5db" containerName="revoke-keys" Nov 01 22:34:46.848721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.848626 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e20e20c-a016-430c-a065-d44acb05e5db" containerName="revoke-keys" Nov 01 22:34:46.852101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.852077 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.866118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.866091 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn"] Nov 01 22:34:46.902091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.902061 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" Nov 01 22:34:46.902091 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.902072 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4" event={"ID":"2e20e20c-a016-430c-a065-d44acb05e5db","Type":"ContainerDied","Data":"559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8"} Nov 01 22:34:46.902254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.902121 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559abb2cd05d8b4394d9c2cf9b479db60d6332879c9e22da0d004ba930bd5be8" Nov 01 22:34:46.913245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913223 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913262 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913286 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913451 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbd4\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-kube-api-access-zzbd4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913512 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913565 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913550 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913584 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913670 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:46.913799 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:46.913722 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.014849 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.014819 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.014849 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.014992 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbd4\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-kube-api-access-zzbd4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015091 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015190 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015237 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015265 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015278 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015335 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015560 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015642 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.015855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.015827 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.016176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.016153 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.017417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.017392 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.017635 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.017618 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.027176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.027156 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.027381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.027363 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbd4\" (UniqueName: \"kubernetes.io/projected/65ac1800-ff60-4b0d-9967-5dc3a169c3d8-kube-api-access-zzbd4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8xkxn\" (UID: \"65ac1800-ff60-4b0d-9967-5dc3a169c3d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.168184 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.168115 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:47.452468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.452318 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn"] Nov 01 22:34:47.454664 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:34:47.454636 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ac1800_ff60_4b0d_9967_5dc3a169c3d8.slice/crio-f15d1167652473dce316f1dd67f950058a2c9bca5395aed85b3781363e4a879b WatchSource:0}: Error finding container f15d1167652473dce316f1dd67f950058a2c9bca5395aed85b3781363e4a879b: Status 404 returned error can't find the container with id f15d1167652473dce316f1dd67f950058a2c9bca5395aed85b3781363e4a879b Nov 01 22:34:47.456853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.456820 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:47.456959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.456893 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:47.456959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.456934 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:34:47.909393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.909356 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" event={"ID":"65ac1800-ff60-4b0d-9967-5dc3a169c3d8","Type":"ContainerStarted","Data":"d7b54f2757756c33e2b38c6262c2e456b70e6632349a4aed8dd7bbb1fc50141f"} Nov 01 22:34:47.909590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.909416 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" event={"ID":"65ac1800-ff60-4b0d-9967-5dc3a169c3d8","Type":"ContainerStarted","Data":"f15d1167652473dce316f1dd67f950058a2c9bca5395aed85b3781363e4a879b"} Nov 01 22:34:47.934584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:47.934537 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podStartSLOduration=1.934519804 podStartE2EDuration="1.934519804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:47.932450424 +0000 UTC m=+3034.906352964" watchObservedRunningTime="2026-11-01 22:34:47.934519804 +0000 UTC m=+3034.908422343" Nov 01 22:34:48.169241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:48.169158 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:34:48.170428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:48.170409 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.170484 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:48.170457 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:49.169381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:49.169346 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.169862 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:49.169405 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:50.168439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:50.168406 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.168629 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:50.168459 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:51.168953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:51.168921 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.169375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:51.168974 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:52.168646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:52.168603 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.168842 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:52.168667 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:52.321450 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:34:52.321418 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:34:53.168763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.168724 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.168973 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.168790 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:53.973694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.973660 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd"] Nov 01 22:34:53.977457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.977437 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:53.986954 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.986928 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:53.987252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.987228 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:34:53.988348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:53.988323 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd"] Nov 01 22:34:54.080978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.080949 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdwr4\" (UniqueName: \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.081171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.080990 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.081171 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.081019 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.169474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.169437 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.169670 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.169490 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Nov 01 22:34:54.182350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.182320 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.182497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.182467 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdwr4\" (UniqueName: \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.182568 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.182515 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.183257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.183159 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.184985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.184961 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.195220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.195174 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdwr4\" (UniqueName: \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\") pod \"maas-api-revoke-sub-ec843018-2d5wd\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.292888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.292862 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:54.573040 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.573000 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd"] Nov 01 22:34:54.575808 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:34:54.575776 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a8b995_480a_4105_9d7d_25c07c8a8522.slice/crio-30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309 WatchSource:0}: Error finding container 30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309: Status 404 returned error can't find the container with id 30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309 Nov 01 22:34:54.949847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.949799 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" event={"ID":"94a8b995-480a-4105-9d7d-25c07c8a8522","Type":"ContainerStarted","Data":"eea6bf865c33c49d46761d095fe3726502ac3706197a9672437f3d257ff1272d"} Nov 01 22:34:54.950017 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.949874 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" event={"ID":"94a8b995-480a-4105-9d7d-25c07c8a8522","Type":"ContainerStarted","Data":"30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309"} Nov 01 22:34:54.970039 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:54.969983 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" podStartSLOduration=1.969969388 podStartE2EDuration="1.969969388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.967750446 +0000 UTC m=+3041.941652985" watchObservedRunningTime="2026-11-01 22:34:54.969969388 +0000 UTC m=+3041.943871965" Nov 01 22:34:55.956617 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:55.956585 4395 generic.go:356] "Generic (PLEG): container finished" podID="94a8b995-480a-4105-9d7d-25c07c8a8522" containerID="eea6bf865c33c49d46761d095fe3726502ac3706197a9672437f3d257ff1272d" exitCode=0 Nov 01 22:34:55.957052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:55.956669 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" event={"ID":"94a8b995-480a-4105-9d7d-25c07c8a8522","Type":"ContainerDied","Data":"eea6bf865c33c49d46761d095fe3726502ac3706197a9672437f3d257ff1272d"} Nov 01 22:34:56.170155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:56.170115 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:56.170312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:56.170182 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:57.169236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.169204 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:57.238097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.213116 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:57.238097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.213171 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:57.324058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.319377 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\" (UniqueName: \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\") pod \"94a8b995-480a-4105-9d7d-25c07c8a8522\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " Nov 01 22:34:57.324058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.319476 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\") pod \"94a8b995-480a-4105-9d7d-25c07c8a8522\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " Nov 01 22:34:57.324058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.319545 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\") pod \"94a8b995-480a-4105-9d7d-25c07c8a8522\" (UID: \"94a8b995-480a-4105-9d7d-25c07c8a8522\") " Nov 01 22:34:57.347062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.344598 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4" pod "94a8b995-480a-4105-9d7d-25c07c8a8522" (UID: "94a8b995-480a-4105-9d7d-25c07c8a8522"). InnerVolumeSpecName "kube-api-access-jdwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:57.347649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.347597 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca" pod "94a8b995-480a-4105-9d7d-25c07c8a8522" (UID: "94a8b995-480a-4105-9d7d-25c07c8a8522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:57.349204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.349174 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca" pod "94a8b995-480a-4105-9d7d-25c07c8a8522" (UID: "94a8b995-480a-4105-9d7d-25c07c8a8522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:57.420857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.420620 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:57.420857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.420665 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdwr4\" (UniqueName: \"kubernetes.io/projected/94a8b995-480a-4105-9d7d-25c07c8a8522-kube-api-access-jdwr4\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:57.420857 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.420684 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a8b995-480a-4105-9d7d-25c07c8a8522-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:57.988392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.988356 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" event={"ID":"94a8b995-480a-4105-9d7d-25c07c8a8522","Type":"ContainerDied","Data":"30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309"} Nov 01 22:34:57.988712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.988693 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd" Nov 01 22:34:57.998202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:57.998176 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f408a76476ba7189be8a70a8583bb4cc0efb42c71130bbc286c98664adb309" Nov 01 22:34:58.226050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:58.225202 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:58.226050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:58.225306 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:59.238907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:59.234220 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:34:59.238907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:34:59.234283 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:35:00.276209 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:00.276167 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:00.276645 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:00.276246 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:00.857227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:00.857191 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:35:00.857653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:00.857629 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:35:00.858473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:00.858455 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" Nov 01 22:35:17.328859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:17.328827 4395 scope.go:122] "RemoveContainer" containerID="3280bf24cc006a94c08fb597620b9ebc7d00bdcb3629ab3701410f76b7c1cbfb" Nov 01 22:35:20.029647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.029612 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl"] Nov 01 22:35:20.030182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.030148 4395 cpu_manager.go:414] "containerMap: removing container" podUID="94a8b995-480a-4105-9d7d-25c07c8a8522" containerName="revoke-keys" Nov 01 22:35:20.030182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.030164 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a8b995-480a-4105-9d7d-25c07c8a8522" containerName="revoke-keys" Nov 01 22:35:20.030288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.030246 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a8b995-480a-4105-9d7d-25c07c8a8522" containerName="revoke-keys" Nov 01 22:35:20.034666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.034644 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.038688 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.038634 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:20.038985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.038823 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:35:20.042366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.042012 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl"] Nov 01 22:35:20.135374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.135341 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8wb\" (UniqueName: \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.135546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.135481 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.135546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.135523 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.236606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.236575 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.236770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.236623 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.236770 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.236673 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8wb\" (UniqueName: \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.237700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.237671 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.240329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.240297 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.249578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.249547 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8wb\" (UniqueName: \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\") pod \"maas-api-revoke-sub-046e1343-wsmzl\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.349225 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.349175 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:20.623117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:20.623089 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl"] Nov 01 22:35:20.625441 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:35:20.625396 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4284d1c_ee57_4f6f_bb3a_5019a50ad46a.slice/crio-8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7 WatchSource:0}: Error finding container 8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7: Status 404 returned error can't find the container with id 8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7 Nov 01 22:35:21.207202 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:21.207165 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" event={"ID":"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a","Type":"ContainerStarted","Data":"c8ff2cd7db9bccd914f718367d7cea03af65adb0cc6cfccbc98774acae0735f0"} Nov 01 22:35:21.207640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:21.207219 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" event={"ID":"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a","Type":"ContainerStarted","Data":"8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7"} Nov 01 22:35:21.227098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:21.227054 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" podStartSLOduration=1.22703973 podStartE2EDuration="1.22703973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:21.224921967 +0000 UTC m=+3068.198824505" watchObservedRunningTime="2026-11-01 22:35:21.22703973 +0000 UTC m=+3068.200942260" Nov 01 22:35:22.214737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:22.214703 4395 generic.go:356] "Generic (PLEG): container finished" podID="d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" containerID="c8ff2cd7db9bccd914f718367d7cea03af65adb0cc6cfccbc98774acae0735f0" exitCode=0 Nov 01 22:35:22.215257 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:22.214758 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" event={"ID":"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a","Type":"ContainerDied","Data":"c8ff2cd7db9bccd914f718367d7cea03af65adb0cc6cfccbc98774acae0735f0"} Nov 01 22:35:22.302511 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:35:22.302478 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:35:23.352374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.352346 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:23.367610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.367587 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\") pod \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " Nov 01 22:35:23.367733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.367653 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\" (UniqueName: \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\") pod \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " Nov 01 22:35:23.367825 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.367808 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\") pod \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\" (UID: \"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a\") " Nov 01 22:35:23.368161 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.368136 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca" pod "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" (UID: "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:23.369744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.369716 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb" pod "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" (UID: "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a"). InnerVolumeSpecName "kube-api-access-cw8wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:23.369846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.369779 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca" pod "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" (UID: "d4284d1c-ee57-4f6f-bb3a-5019a50ad46a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:23.468923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.468876 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:23.468923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.468916 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8wb\" (UniqueName: \"kubernetes.io/projected/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-kube-api-access-cw8wb\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:23.468923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:23.468929 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:24.226938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:24.226896 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" event={"ID":"d4284d1c-ee57-4f6f-bb3a-5019a50ad46a","Type":"ContainerDied","Data":"8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7"} Nov 01 22:35:24.226938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:24.226915 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl" Nov 01 22:35:24.226938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:24.226945 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d96713d6c5d221322c8e6380f69a0a80de7e22dd7c82e27319cdbc305d1b0c7" Nov 01 22:35:30.278705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.278674 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8"] Nov 01 22:35:30.281173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.279079 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" containerName="revoke-keys" Nov 01 22:35:30.281173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.279093 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" containerName="revoke-keys" Nov 01 22:35:30.281173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.279193 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" containerName="revoke-keys" Nov 01 22:35:30.282166 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.282149 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.288510 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.288485 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:30.290334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.290297 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:35:30.294514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.294495 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8"] Nov 01 22:35:30.328181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.328147 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tzr\" (UniqueName: \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.328315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.328260 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.328442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.328421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.429253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.429218 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.429441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.429269 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tzr\" (UniqueName: \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.429441 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.429317 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.430020 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.429999 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.431773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.431750 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.442052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.442000 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tzr\" (UniqueName: \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\") pod \"maas-api-revoke-sub-d3fba056-7xtl8\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.598498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.598428 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:30.874169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:30.874146 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8"] Nov 01 22:35:30.875828 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:35:30.875791 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f33839_441f_449d_b10c_adc4218aa351.slice/crio-377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601 WatchSource:0}: Error finding container 377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601: Status 404 returned error can't find the container with id 377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601 Nov 01 22:35:31.269778 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:31.269744 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" event={"ID":"45f33839-441f-449d-b10c-adc4218aa351","Type":"ContainerStarted","Data":"15d8cefc04bd8b308dd3a5c9560e1970e3e9ff635c244613e9b6e3864cb3c14f"} Nov 01 22:35:31.269965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:31.269804 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" event={"ID":"45f33839-441f-449d-b10c-adc4218aa351","Type":"ContainerStarted","Data":"377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601"} Nov 01 22:35:31.287668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:31.287510 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" podStartSLOduration=1.287494063 podStartE2EDuration="1.287494063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:31.286750568 +0000 UTC m=+3078.260653095" watchObservedRunningTime="2026-11-01 22:35:31.287494063 +0000 UTC m=+3078.261396604" Nov 01 22:35:32.286064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:32.280959 4395 generic.go:356] "Generic (PLEG): container finished" podID="45f33839-441f-449d-b10c-adc4218aa351" containerID="15d8cefc04bd8b308dd3a5c9560e1970e3e9ff635c244613e9b6e3864cb3c14f" exitCode=0 Nov 01 22:35:32.286064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:32.281207 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" event={"ID":"45f33839-441f-449d-b10c-adc4218aa351","Type":"ContainerDied","Data":"15d8cefc04bd8b308dd3a5c9560e1970e3e9ff635c244613e9b6e3864cb3c14f"} Nov 01 22:35:33.997179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:33.997149 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:34.045532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.045492 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\") pod \"45f33839-441f-449d-b10c-adc4218aa351\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " Nov 01 22:35:34.045721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.045620 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\") pod \"45f33839-441f-449d-b10c-adc4218aa351\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " Nov 01 22:35:34.045808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.045764 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\" (UniqueName: \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\") pod \"45f33839-441f-449d-b10c-adc4218aa351\" (UID: \"45f33839-441f-449d-b10c-adc4218aa351\") " Nov 01 22:35:34.045920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.045892 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca" pod "45f33839-441f-449d-b10c-adc4218aa351" (UID: "45f33839-441f-449d-b10c-adc4218aa351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:34.046118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.046100 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f33839-441f-449d-b10c-adc4218aa351-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:34.048124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.048095 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr" pod "45f33839-441f-449d-b10c-adc4218aa351" (UID: "45f33839-441f-449d-b10c-adc4218aa351"). InnerVolumeSpecName "kube-api-access-q6tzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:34.048657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.048630 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca" pod "45f33839-441f-449d-b10c-adc4218aa351" (UID: "45f33839-441f-449d-b10c-adc4218aa351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:34.146717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.146688 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6tzr\" (UniqueName: \"kubernetes.io/projected/45f33839-441f-449d-b10c-adc4218aa351-kube-api-access-q6tzr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:34.146717 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.146712 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f33839-441f-449d-b10c-adc4218aa351-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:34.302432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.302395 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" Nov 01 22:35:34.302432 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.302404 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8" event={"ID":"45f33839-441f-449d-b10c-adc4218aa351","Type":"ContainerDied","Data":"377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601"} Nov 01 22:35:34.302660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:34.302459 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377c9966cf6a08eb80c8c94d57cd207ef02673707a45f51cd4cf45086e176601" Nov 01 22:35:40.028553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:40.028517 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt"] Nov 01 22:35:40.030814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:40.030787 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c95f262-hwjvt"] Nov 01 22:35:41.162792 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:41.162762 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d" path="/var/lib/kubelet/pods/b46f8a42-4b1a-4f1a-ad7c-765a9be08c3d/volumes" Nov 01 22:35:50.829639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.829604 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv"] Nov 01 22:35:50.830153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.830095 4395 cpu_manager.go:414] "containerMap: removing container" podUID="45f33839-441f-449d-b10c-adc4218aa351" containerName="revoke-keys" Nov 01 22:35:50.830153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.830110 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f33839-441f-449d-b10c-adc4218aa351" containerName="revoke-keys" Nov 01 22:35:50.830232 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.830185 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f33839-441f-449d-b10c-adc4218aa351" containerName="revoke-keys" Nov 01 22:35:50.833455 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.833440 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:50.839350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.839327 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:50.839865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.839840 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:35:50.844057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.844016 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv"] Nov 01 22:35:50.967889 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.967862 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jtq\" (UniqueName: \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:50.968053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.967911 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:50.968053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:50.968003 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.068757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.068728 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jtq\" (UniqueName: \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.068897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.068780 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.068897 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.068816 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.069428 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.069403 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.071397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.071370 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.081396 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.081339 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jtq\" (UniqueName: \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\") pod \"maas-api-revoke-sub-ae594979-c7hpv\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.148156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.148133 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:51.425647 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.425618 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv"] Nov 01 22:35:51.427306 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:35:51.427277 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ff2446_c6ad_4734_86ff_af58f420a517.slice/crio-0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196 WatchSource:0}: Error finding container 0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196: Status 404 returned error can't find the container with id 0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196 Nov 01 22:35:51.453272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:51.453242 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" event={"ID":"98ff2446-c6ad-4734-86ff-af58f420a517","Type":"ContainerStarted","Data":"0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196"} Nov 01 22:35:52.300115 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:35:52.300076 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:35:52.460409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:52.460331 4395 generic.go:356] "Generic (PLEG): container finished" podID="98ff2446-c6ad-4734-86ff-af58f420a517" containerID="0de0da16748adcd34e97e67584e8b3e5ae8e0cd97f2646e607ca8d36577f148d" exitCode=0 Nov 01 22:35:52.460559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:52.460467 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" event={"ID":"98ff2446-c6ad-4734-86ff-af58f420a517","Type":"ContainerDied","Data":"0de0da16748adcd34e97e67584e8b3e5ae8e0cd97f2646e607ca8d36577f148d"} Nov 01 22:35:53.596285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.596256 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:53.693293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.693265 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\" (UniqueName: \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\") pod \"98ff2446-c6ad-4734-86ff-af58f420a517\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " Nov 01 22:35:53.693439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.693322 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\") pod \"98ff2446-c6ad-4734-86ff-af58f420a517\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " Nov 01 22:35:53.693439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.693360 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\") pod \"98ff2446-c6ad-4734-86ff-af58f420a517\" (UID: \"98ff2446-c6ad-4734-86ff-af58f420a517\") " Nov 01 22:35:53.693793 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.693760 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca" pod "98ff2446-c6ad-4734-86ff-af58f420a517" (UID: "98ff2446-c6ad-4734-86ff-af58f420a517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:53.695424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.695396 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq" pod "98ff2446-c6ad-4734-86ff-af58f420a517" (UID: "98ff2446-c6ad-4734-86ff-af58f420a517"). InnerVolumeSpecName "kube-api-access-58jtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:53.695508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.695485 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca" pod "98ff2446-c6ad-4734-86ff-af58f420a517" (UID: "98ff2446-c6ad-4734-86ff-af58f420a517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:53.794402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.794369 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jtq\" (UniqueName: \"kubernetes.io/projected/98ff2446-c6ad-4734-86ff-af58f420a517-kube-api-access-58jtq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:53.794402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.794397 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:53.794575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:53.794410 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ff2446-c6ad-4734-86ff-af58f420a517-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:54.472306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:54.472221 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" Nov 01 22:35:54.472306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:54.472254 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv" event={"ID":"98ff2446-c6ad-4734-86ff-af58f420a517","Type":"ContainerDied","Data":"0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196"} Nov 01 22:35:54.472306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:35:54.472303 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c3f1652151af94011c455fd7a0d57b3c54ef17a0db7a3745fcd2f8fe2249196" Nov 01 22:36:02.023247 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:02.023212 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f"] Nov 01 22:36:02.029917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:02.029887 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed035fcf-lj56f"] Nov 01 22:36:03.163203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:03.163170 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b035eaf-86fc-42bc-86fa-3d4def81625a" path="/var/lib/kubelet/pods/6b035eaf-86fc-42bc-86fa-3d4def81625a/volumes" Nov 01 22:36:08.022409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.022370 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw"] Nov 01 22:36:08.025147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.025124 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cd5e8d4-xv7fw"] Nov 01 22:36:08.806045 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.805990 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq"] Nov 01 22:36:08.806461 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.806442 4395 cpu_manager.go:414] "containerMap: removing container" podUID="98ff2446-c6ad-4734-86ff-af58f420a517" containerName="revoke-keys" Nov 01 22:36:08.806559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.806464 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ff2446-c6ad-4734-86ff-af58f420a517" containerName="revoke-keys" Nov 01 22:36:08.806623 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.806592 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ff2446-c6ad-4734-86ff-af58f420a517" containerName="revoke-keys" Nov 01 22:36:08.810969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.810949 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.814899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.814878 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:08.816053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.816012 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:36:08.819464 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.819343 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq"] Nov 01 22:36:08.880761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.880734 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.880888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.880794 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.880888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.880828 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjfq\" (UniqueName: \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.982114 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.982085 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjfq\" (UniqueName: \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.982239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.982142 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.982239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.982192 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.982831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.982812 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.984569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.984547 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:08.994524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:08.994507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjfq\" (UniqueName: \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\") pod \"maas-api-revoke-sub-a4aef7d7-jghjq\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:09.127341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:09.127272 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:09.163859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:09.163835 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02353b4a-6603-4af8-8f49-161b455410c9" path="/var/lib/kubelet/pods/02353b4a-6603-4af8-8f49-161b455410c9/volumes" Nov 01 22:36:09.402706 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:09.402679 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq"] Nov 01 22:36:09.404875 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:36:09.404847 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda32023e_2310_47e6_90f9_76e97e0431a4.slice/crio-394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663 WatchSource:0}: Error finding container 394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663: Status 404 returned error can't find the container with id 394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663 Nov 01 22:36:09.578531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:09.578498 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" event={"ID":"da32023e-2310-47e6-90f9-76e97e0431a4","Type":"ContainerStarted","Data":"394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663"} Nov 01 22:36:10.027899 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:10.027868 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h"] Nov 01 22:36:10.031702 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:10.031675 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af18be7-mzr4h"] Nov 01 22:36:10.585473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:10.585439 4395 generic.go:356] "Generic (PLEG): container finished" podID="da32023e-2310-47e6-90f9-76e97e0431a4" containerID="6cfc54f819a389bf7224c6911d90a4806583830db3a750c84bb6d998d48da3bd" exitCode=0 Nov 01 22:36:10.585827 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:10.585481 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" event={"ID":"da32023e-2310-47e6-90f9-76e97e0431a4","Type":"ContainerDied","Data":"6cfc54f819a389bf7224c6911d90a4806583830db3a750c84bb6d998d48da3bd"} Nov 01 22:36:11.163390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.163355 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca5bc60-4778-4bca-be96-3ff971cde016" path="/var/lib/kubelet/pods/eca5bc60-4778-4bca-be96-3ff971cde016/volumes" Nov 01 22:36:11.727378 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.727355 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:11.805722 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.805694 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\") pod \"da32023e-2310-47e6-90f9-76e97e0431a4\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " Nov 01 22:36:11.805886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.805797 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\" (UniqueName: \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\") pod \"da32023e-2310-47e6-90f9-76e97e0431a4\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " Nov 01 22:36:11.805886 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.805840 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\") pod \"da32023e-2310-47e6-90f9-76e97e0431a4\" (UID: \"da32023e-2310-47e6-90f9-76e97e0431a4\") " Nov 01 22:36:11.806147 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.806125 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca" pod "da32023e-2310-47e6-90f9-76e97e0431a4" (UID: "da32023e-2310-47e6-90f9-76e97e0431a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:11.808111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.808087 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca" pod "da32023e-2310-47e6-90f9-76e97e0431a4" (UID: "da32023e-2310-47e6-90f9-76e97e0431a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:11.808211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.808115 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq" pod "da32023e-2310-47e6-90f9-76e97e0431a4" (UID: "da32023e-2310-47e6-90f9-76e97e0431a4"). InnerVolumeSpecName "kube-api-access-hvjfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:11.907315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.907264 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjfq\" (UniqueName: \"kubernetes.io/projected/da32023e-2310-47e6-90f9-76e97e0431a4-kube-api-access-hvjfq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:11.907315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.907287 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:11.907315 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:11.907296 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da32023e-2310-47e6-90f9-76e97e0431a4-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:12.597690 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:12.597663 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" Nov 01 22:36:12.597873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:12.597707 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq" event={"ID":"da32023e-2310-47e6-90f9-76e97e0431a4","Type":"ContainerDied","Data":"394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663"} Nov 01 22:36:12.597873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:12.597763 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="394b4bc104dbedcbc05af50696134dd2ac8902df48cad82fc00da52527ce7663" Nov 01 22:36:22.315717 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:36:22.315670 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:36:27.598252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.598215 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6"] Nov 01 22:36:27.598774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.598596 4395 cpu_manager.go:414] "containerMap: removing container" podUID="da32023e-2310-47e6-90f9-76e97e0431a4" containerName="revoke-keys" Nov 01 22:36:27.598774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.598607 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da32023e-2310-47e6-90f9-76e97e0431a4" containerName="revoke-keys" Nov 01 22:36:27.598774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.598688 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="da32023e-2310-47e6-90f9-76e97e0431a4" containerName="revoke-keys" Nov 01 22:36:27.603138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.603115 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.607477 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.607457 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:27.608571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.608550 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:36:27.612916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.612894 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6"] Nov 01 22:36:27.688413 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.688386 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.688594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.688421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.688594 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.688476 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.789176 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.789148 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.789365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.789180 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.789365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.789230 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.789831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.789807 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.791714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.791691 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.801733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.801714 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\") pod \"maas-api-revoke-sub-79805ca9-wtfr6\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:27.919865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:27.919794 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:28.196350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:28.196117 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6"] Nov 01 22:36:28.199214 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:36:28.199182 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48bca65_63ba_44e4_bdd0_2346849a7193.slice/crio-a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125 WatchSource:0}: Error finding container a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125: Status 404 returned error can't find the container with id a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125 Nov 01 22:36:28.709154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:28.709117 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" event={"ID":"b48bca65-63ba-44e4-bdd0-2346849a7193","Type":"ContainerStarted","Data":"2fde8f48665a24e75104c3de5fb69e371a5e459c41ba543ec166e01ad830c7a9"} Nov 01 22:36:28.709625 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:28.709164 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" event={"ID":"b48bca65-63ba-44e4-bdd0-2346849a7193","Type":"ContainerStarted","Data":"a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125"} Nov 01 22:36:28.729043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:28.728974 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" podStartSLOduration=1.728961014 podStartE2EDuration="1.728961014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:28.725936548 +0000 UTC m=+3135.699839087" watchObservedRunningTime="2026-11-01 22:36:28.728961014 +0000 UTC m=+3135.702863563" Nov 01 22:36:29.715864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:29.715830 4395 generic.go:356] "Generic (PLEG): container finished" podID="b48bca65-63ba-44e4-bdd0-2346849a7193" containerID="2fde8f48665a24e75104c3de5fb69e371a5e459c41ba543ec166e01ad830c7a9" exitCode=0 Nov 01 22:36:29.716286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:29.715946 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" event={"ID":"b48bca65-63ba-44e4-bdd0-2346849a7193","Type":"ContainerDied","Data":"2fde8f48665a24e75104c3de5fb69e371a5e459c41ba543ec166e01ad830c7a9"} Nov 01 22:36:30.854529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.854504 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:30.918576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.918549 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\") pod \"b48bca65-63ba-44e4-bdd0-2346849a7193\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " Nov 01 22:36:30.918774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.918634 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\") pod \"b48bca65-63ba-44e4-bdd0-2346849a7193\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " Nov 01 22:36:30.918774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.918671 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\") pod \"b48bca65-63ba-44e4-bdd0-2346849a7193\" (UID: \"b48bca65-63ba-44e4-bdd0-2346849a7193\") " Nov 01 22:36:30.919098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.919075 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca" pod "b48bca65-63ba-44e4-bdd0-2346849a7193" (UID: "b48bca65-63ba-44e4-bdd0-2346849a7193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:30.920833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.920810 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca" pod "b48bca65-63ba-44e4-bdd0-2346849a7193" (UID: "b48bca65-63ba-44e4-bdd0-2346849a7193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:30.920953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:30.920812 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv" pod "b48bca65-63ba-44e4-bdd0-2346849a7193" (UID: "b48bca65-63ba-44e4-bdd0-2346849a7193"). InnerVolumeSpecName "kube-api-access-thzbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:31.019475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.019445 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:31.019475 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.019476 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48bca65-63ba-44e4-bdd0-2346849a7193-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:31.019659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.019488 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzbv\" (UniqueName: \"kubernetes.io/projected/b48bca65-63ba-44e4-bdd0-2346849a7193-kube-api-access-thzbv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:31.728334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.728305 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" Nov 01 22:36:31.728531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.728307 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6" event={"ID":"b48bca65-63ba-44e4-bdd0-2346849a7193","Type":"ContainerDied","Data":"a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125"} Nov 01 22:36:31.728531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:31.728415 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59c19501a63d7192a327e3f9756ae385e6d9bef5b878b036560764d0ba44125" Nov 01 22:36:45.273126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.273094 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5"] Nov 01 22:36:45.273533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.273481 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b48bca65-63ba-44e4-bdd0-2346849a7193" containerName="revoke-keys" Nov 01 22:36:45.273533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.273492 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48bca65-63ba-44e4-bdd0-2346849a7193" containerName="revoke-keys" Nov 01 22:36:45.273615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.273567 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48bca65-63ba-44e4-bdd0-2346849a7193" containerName="revoke-keys" Nov 01 22:36:45.276579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.276562 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.281233 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.281211 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:45.281368 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.281265 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:36:45.287996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.287962 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5"] Nov 01 22:36:45.394633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.394601 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpcm\" (UniqueName: \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.394822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.394670 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.394868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.394821 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.495966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.495937 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpcm\" (UniqueName: \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.496165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.496001 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.496165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.496087 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.496747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.496728 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.498343 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.498324 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.508517 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.508496 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpcm\" (UniqueName: \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\") pod \"maas-api-revoke-sub-0b7e8edb-vvbv5\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.594606 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.594556 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:45.870525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.870500 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5"] Nov 01 22:36:45.872359 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:36:45.872328 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b42e10e_0319_46f1_9835_ce0bb45211c8.slice/crio-c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c WatchSource:0}: Error finding container c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c: Status 404 returned error can't find the container with id c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c Nov 01 22:36:45.915988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:45.915960 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" event={"ID":"5b42e10e-0319-46f1-9835-ce0bb45211c8","Type":"ContainerStarted","Data":"c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c"} Nov 01 22:36:46.923289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:46.923213 4395 generic.go:356] "Generic (PLEG): container finished" podID="5b42e10e-0319-46f1-9835-ce0bb45211c8" containerID="a49298087d92a9c89b1e07df04a29eb552160154f450883d7be3393ae24426dd" exitCode=0 Nov 01 22:36:46.923289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:46.923280 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" event={"ID":"5b42e10e-0319-46f1-9835-ce0bb45211c8","Type":"ContainerDied","Data":"a49298087d92a9c89b1e07df04a29eb552160154f450883d7be3393ae24426dd"} Nov 01 22:36:48.065339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.065312 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:48.221265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.221198 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\") pod \"5b42e10e-0319-46f1-9835-ce0bb45211c8\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " Nov 01 22:36:48.221395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.221269 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\") pod \"5b42e10e-0319-46f1-9835-ce0bb45211c8\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " Nov 01 22:36:48.221395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.221344 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\" (UniqueName: \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\") pod \"5b42e10e-0319-46f1-9835-ce0bb45211c8\" (UID: \"5b42e10e-0319-46f1-9835-ce0bb45211c8\") " Nov 01 22:36:48.221659 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.221632 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca" pod "5b42e10e-0319-46f1-9835-ce0bb45211c8" (UID: "5b42e10e-0319-46f1-9835-ce0bb45211c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:48.223521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.223487 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm" pod "5b42e10e-0319-46f1-9835-ce0bb45211c8" (UID: "5b42e10e-0319-46f1-9835-ce0bb45211c8"). InnerVolumeSpecName "kube-api-access-wmpcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:48.223614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.223526 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca" pod "5b42e10e-0319-46f1-9835-ce0bb45211c8" (UID: "5b42e10e-0319-46f1-9835-ce0bb45211c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:48.322435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.322410 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:48.322435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.322434 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b42e10e-0319-46f1-9835-ce0bb45211c8-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:48.322585 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.322445 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpcm\" (UniqueName: \"kubernetes.io/projected/5b42e10e-0319-46f1-9835-ce0bb45211c8-kube-api-access-wmpcm\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:48.935223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.935193 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" Nov 01 22:36:48.935412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.935194 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5" event={"ID":"5b42e10e-0319-46f1-9835-ce0bb45211c8","Type":"ContainerDied","Data":"c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c"} Nov 01 22:36:48.935412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:48.935320 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f486df3eee55191ce92b886758305c8a543a13d3e3489b2db6bcf16ab2158c" Nov 01 22:36:52.300146 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:36:52.300108 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:36:55.910977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.910939 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb"] Nov 01 22:36:55.911493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.911360 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5b42e10e-0319-46f1-9835-ce0bb45211c8" containerName="revoke-keys" Nov 01 22:36:55.911493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.911372 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b42e10e-0319-46f1-9835-ce0bb45211c8" containerName="revoke-keys" Nov 01 22:36:55.911493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.911467 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b42e10e-0319-46f1-9835-ce0bb45211c8" containerName="revoke-keys" Nov 01 22:36:55.914787 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.914765 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:55.925415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.925388 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb"] Nov 01 22:36:55.926303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.926274 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:55.926303 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.926298 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:36:55.987656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.987620 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:55.987919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.987668 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7rv\" (UniqueName: \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:55.987919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:55.987689 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.088369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.088334 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.088502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.088387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7rv\" (UniqueName: \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.088502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.088414 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.088996 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.088970 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.090844 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.090819 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.100423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.100400 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7rv\" (UniqueName: \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\") pod \"maas-api-revoke-sub-3d4156f1-7sskb\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.233007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.232942 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:56.510832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.510809 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb"] Nov 01 22:36:56.512717 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:36:56.512689 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19406e0_1050_48b6_bd10_5f6ada24023e.slice/crio-cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0 WatchSource:0}: Error finding container cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0: Status 404 returned error can't find the container with id cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0 Nov 01 22:36:56.985698 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.985667 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" event={"ID":"e19406e0-1050-48b6-bd10-5f6ada24023e","Type":"ContainerStarted","Data":"73c002ca769f15eafbb2756becf1efcd20994ef39c9ea212bf91195f01609f2b"} Nov 01 22:36:56.986063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:56.985728 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" event={"ID":"e19406e0-1050-48b6-bd10-5f6ada24023e","Type":"ContainerStarted","Data":"cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0"} Nov 01 22:36:57.005306 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:57.005256 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" podStartSLOduration=2.005241753 podStartE2EDuration="2.005241753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:57.004121997 +0000 UTC m=+3163.978024535" watchObservedRunningTime="2026-11-01 22:36:57.005241753 +0000 UTC m=+3163.979144292" Nov 01 22:36:57.992982 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:57.992943 4395 generic.go:356] "Generic (PLEG): container finished" podID="e19406e0-1050-48b6-bd10-5f6ada24023e" containerID="73c002ca769f15eafbb2756becf1efcd20994ef39c9ea212bf91195f01609f2b" exitCode=0 Nov 01 22:36:57.993393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:57.993045 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" event={"ID":"e19406e0-1050-48b6-bd10-5f6ada24023e","Type":"ContainerDied","Data":"73c002ca769f15eafbb2756becf1efcd20994ef39c9ea212bf91195f01609f2b"} Nov 01 22:36:59.136859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.136833 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:36:59.218776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.218750 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\" (UniqueName: \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\") pod \"e19406e0-1050-48b6-bd10-5f6ada24023e\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " Nov 01 22:36:59.218923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.218795 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\") pod \"e19406e0-1050-48b6-bd10-5f6ada24023e\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " Nov 01 22:36:59.218923 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.218907 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\") pod \"e19406e0-1050-48b6-bd10-5f6ada24023e\" (UID: \"e19406e0-1050-48b6-bd10-5f6ada24023e\") " Nov 01 22:36:59.219302 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.219246 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca" pod "e19406e0-1050-48b6-bd10-5f6ada24023e" (UID: "e19406e0-1050-48b6-bd10-5f6ada24023e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:59.219962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.219926 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:59.221224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.221198 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca" pod "e19406e0-1050-48b6-bd10-5f6ada24023e" (UID: "e19406e0-1050-48b6-bd10-5f6ada24023e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:59.221347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.221330 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv" pod "e19406e0-1050-48b6-bd10-5f6ada24023e" (UID: "e19406e0-1050-48b6-bd10-5f6ada24023e"). InnerVolumeSpecName "kube-api-access-vg7rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:59.320802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.320772 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19406e0-1050-48b6-bd10-5f6ada24023e-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:59.320802 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:36:59.320799 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7rv\" (UniqueName: \"kubernetes.io/projected/e19406e0-1050-48b6-bd10-5f6ada24023e-kube-api-access-vg7rv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:00.005935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:00.005900 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" event={"ID":"e19406e0-1050-48b6-bd10-5f6ada24023e","Type":"ContainerDied","Data":"cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0"} Nov 01 22:37:00.005935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:00.005918 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb" Nov 01 22:37:00.006170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:00.005947 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf4a3e5fdaa0d70211ee00fbb24c54d7325c8e85117b801404784a8bfcada5b0" Nov 01 22:37:11.023593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.023559 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8"] Nov 01 22:37:11.028380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.028351 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62"] Nov 01 22:37:11.030935 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.030910 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a17322f-t4dc8"] Nov 01 22:37:11.032984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.032956 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb28f4a-dlj62"] Nov 01 22:37:11.162898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.162870 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd862bd7-be6a-43cc-9368-9220366b33ce" path="/var/lib/kubelet/pods/bd862bd7-be6a-43cc-9368-9220366b33ce/volumes" Nov 01 22:37:11.163317 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:11.163303 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a9c9b9-5f31-499a-9ad9-3b4203fabce6" path="/var/lib/kubelet/pods/e1a9c9b9-5f31-499a-9ad9-3b4203fabce6/volumes" Nov 01 22:37:13.872543 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.872497 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb"] Nov 01 22:37:13.878193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.878162 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7sskb"] Nov 01 22:37:13.898245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.898213 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j"] Nov 01 22:37:13.898682 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.898669 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e19406e0-1050-48b6-bd10-5f6ada24023e" containerName="revoke-keys" Nov 01 22:37:13.898727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.898683 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19406e0-1050-48b6-bd10-5f6ada24023e" containerName="revoke-keys" Nov 01 22:37:13.898784 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.898774 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19406e0-1050-48b6-bd10-5f6ada24023e" containerName="revoke-keys" Nov 01 22:37:13.901929 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.901910 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:13.909092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.909053 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:13.909211 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.909116 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:37:13.911589 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:13.911567 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j"] Nov 01 22:37:14.003174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.003139 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.003336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.003187 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.003336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.003294 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hccl\" (UniqueName: \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.103771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.103739 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hccl\" (UniqueName: \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.103993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.103823 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.103993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.103859 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.104470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.104449 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.106503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.106478 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.115930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.115909 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hccl\" (UniqueName: \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\") pod \"maas-api-revoke-sub-3d4156f1-zj44j\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.217509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.217428 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:14.495553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:14.495525 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j"] Nov 01 22:37:14.497084 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:37:14.497058 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f40084_e5c0_4119_b2d0_abbba80c48c8.slice/crio-89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9 WatchSource:0}: Error finding container 89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9: Status 404 returned error can't find the container with id 89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9 Nov 01 22:37:15.114750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:15.114645 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" event={"ID":"28f40084-e5c0-4119-b2d0-abbba80c48c8","Type":"ContainerStarted","Data":"fac654319a67a0e49aa8f7a1cf0fcfa233928bb938e606afdd9476d234155cb2"} Nov 01 22:37:15.114750 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:15.114713 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" event={"ID":"28f40084-e5c0-4119-b2d0-abbba80c48c8","Type":"ContainerStarted","Data":"89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9"} Nov 01 22:37:15.135143 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:15.135095 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" podStartSLOduration=2.135077527 podStartE2EDuration="2.135077527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:15.132869012 +0000 UTC m=+3182.106771550" watchObservedRunningTime="2026-11-01 22:37:15.135077527 +0000 UTC m=+3182.108980250" Nov 01 22:37:15.162671 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:15.162648 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19406e0-1050-48b6-bd10-5f6ada24023e" path="/var/lib/kubelet/pods/e19406e0-1050-48b6-bd10-5f6ada24023e/volumes" Nov 01 22:37:16.022746 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.022710 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm"] Nov 01 22:37:16.028108 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.028082 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq"] Nov 01 22:37:16.030347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.030331 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e230933-5b2pm"] Nov 01 22:37:16.032337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.032312 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb6a478-kpwsq"] Nov 01 22:37:16.122063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.122017 4395 generic.go:356] "Generic (PLEG): container finished" podID="28f40084-e5c0-4119-b2d0-abbba80c48c8" containerID="fac654319a67a0e49aa8f7a1cf0fcfa233928bb938e606afdd9476d234155cb2" exitCode=0 Nov 01 22:37:16.122412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:16.122058 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" event={"ID":"28f40084-e5c0-4119-b2d0-abbba80c48c8","Type":"ContainerDied","Data":"fac654319a67a0e49aa8f7a1cf0fcfa233928bb938e606afdd9476d234155cb2"} Nov 01 22:37:17.163538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.163503 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ab6af0-91ca-4396-a1e1-fe588b28e3c5" path="/var/lib/kubelet/pods/22ab6af0-91ca-4396-a1e1-fe588b28e3c5/volumes" Nov 01 22:37:17.163963 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.163948 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be701dd7-3026-4129-970d-8a0dbca73b67" path="/var/lib/kubelet/pods/be701dd7-3026-4129-970d-8a0dbca73b67/volumes" Nov 01 22:37:17.262829 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.262802 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:17.332990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.332960 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\") pod \"28f40084-e5c0-4119-b2d0-abbba80c48c8\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " Nov 01 22:37:17.333181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.333105 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\" (UniqueName: \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\") pod \"28f40084-e5c0-4119-b2d0-abbba80c48c8\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " Nov 01 22:37:17.333181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.333149 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\") pod \"28f40084-e5c0-4119-b2d0-abbba80c48c8\" (UID: \"28f40084-e5c0-4119-b2d0-abbba80c48c8\") " Nov 01 22:37:17.333327 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.333304 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca" pod "28f40084-e5c0-4119-b2d0-abbba80c48c8" (UID: "28f40084-e5c0-4119-b2d0-abbba80c48c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:17.333468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.333452 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:17.335365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.335342 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca" pod "28f40084-e5c0-4119-b2d0-abbba80c48c8" (UID: "28f40084-e5c0-4119-b2d0-abbba80c48c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:17.335542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.335404 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl" pod "28f40084-e5c0-4119-b2d0-abbba80c48c8" (UID: "28f40084-e5c0-4119-b2d0-abbba80c48c8"). InnerVolumeSpecName "kube-api-access-5hccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:17.360133 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.360110 4395 scope.go:122] "RemoveContainer" containerID="28416cd3eaf4e467776798bd9b03361ac8217cd0de60de72877332c3c362cfa5" Nov 01 22:37:17.369180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.369162 4395 scope.go:122] "RemoveContainer" containerID="c3ad621c19a1759df28244e3bcb47a2e4364d62b8f4879685dd4a3b9e0fb29ec" Nov 01 22:37:17.376964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.376946 4395 scope.go:122] "RemoveContainer" containerID="7cbea1bfbc500151387d7a03bbb873f4992841fdf4587e2b5c70782c17195f58" Nov 01 22:37:17.384516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.384499 4395 scope.go:122] "RemoveContainer" containerID="2c5a5338d53dc18d8ca04c87223545468f18223075872979a12c63e4bb7e4ef5" Nov 01 22:37:17.392504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.392484 4395 scope.go:122] "RemoveContainer" containerID="9365fd494a926262389d2ce951aaecd698a0af2d035a1cd8c6f0fba54d1a96d0" Nov 01 22:37:17.400265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.400249 4395 scope.go:122] "RemoveContainer" containerID="9631efd6d024e4b4b5e52ee8edfef5ee02443b036999dc828ca63d24e7456a50" Nov 01 22:37:17.407832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.407816 4395 scope.go:122] "RemoveContainer" containerID="b474681855f969b7daef1a158f1ed9bc0e772dd9f1301f9eed19b6890ae5c36c" Nov 01 22:37:17.434131 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.434113 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hccl\" (UniqueName: \"kubernetes.io/projected/28f40084-e5c0-4119-b2d0-abbba80c48c8-kube-api-access-5hccl\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:17.434193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:17.434134 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f40084-e5c0-4119-b2d0-abbba80c48c8-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:18.135409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:18.135377 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" Nov 01 22:37:18.135409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:18.135403 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j" event={"ID":"28f40084-e5c0-4119-b2d0-abbba80c48c8","Type":"ContainerDied","Data":"89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9"} Nov 01 22:37:18.135639 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:18.135454 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89572791dcee14bcf0432790ecbe3d11e04cc34856a9c96b52a3987799f4bed9" Nov 01 22:37:22.300689 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:37:22.300654 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:37:42.147253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.147162 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv"] Nov 01 22:37:42.148082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.147601 4395 cpu_manager.go:414] "containerMap: removing container" podUID="28f40084-e5c0-4119-b2d0-abbba80c48c8" containerName="revoke-keys" Nov 01 22:37:42.148082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.147616 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f40084-e5c0-4119-b2d0-abbba80c48c8" containerName="revoke-keys" Nov 01 22:37:42.148082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.147697 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f40084-e5c0-4119-b2d0-abbba80c48c8" containerName="revoke-keys" Nov 01 22:37:42.153096 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.153076 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.158117 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.158091 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:37:42.158255 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.158132 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:42.159612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.159592 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv"] Nov 01 22:37:42.298838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.298799 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.299081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.299057 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqd5\" (UniqueName: \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.299253 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.299235 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.400728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.400644 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.400728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.400719 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.400965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.400764 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqd5\" (UniqueName: \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.401473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.401454 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.403210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.403190 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.412966 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.412939 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqd5\" (UniqueName: \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\") pod \"maas-api-revoke-sub-e2fc9c5b-x2ppv\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.469079 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.469059 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:42.747445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:42.747416 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv"] Nov 01 22:37:42.748900 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:37:42.748869 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8462d1df_2ed7_48fa_a7f6_dc4ae19e035f.slice/crio-21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd WatchSource:0}: Error finding container 21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd: Status 404 returned error can't find the container with id 21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd Nov 01 22:37:43.309530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:43.309494 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" event={"ID":"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f","Type":"ContainerStarted","Data":"8d26ee82da235e97e57def95f43b021d6d145e278b17ed817b730d89532946f3"} Nov 01 22:37:43.309925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:43.309548 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" event={"ID":"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f","Type":"ContainerStarted","Data":"21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd"} Nov 01 22:37:43.332010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:43.331923 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" podStartSLOduration=1.331911656 podStartE2EDuration="1.331911656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:43.330640393 +0000 UTC m=+3210.304542920" watchObservedRunningTime="2026-11-01 22:37:43.331911656 +0000 UTC m=+3210.305814195" Nov 01 22:37:44.323674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:44.323637 4395 generic.go:356] "Generic (PLEG): container finished" podID="8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" containerID="8d26ee82da235e97e57def95f43b021d6d145e278b17ed817b730d89532946f3" exitCode=0 Nov 01 22:37:44.324126 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:44.323719 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" event={"ID":"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f","Type":"ContainerDied","Data":"8d26ee82da235e97e57def95f43b021d6d145e278b17ed817b730d89532946f3"} Nov 01 22:37:45.468654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.468628 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:45.533223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.533192 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\" (UniqueName: \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\") pod \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " Nov 01 22:37:45.533393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.533238 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\") pod \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " Nov 01 22:37:45.533393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.533264 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\") pod \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\" (UID: \"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f\") " Nov 01 22:37:45.533674 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.533648 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca" pod "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" (UID: "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:45.535401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.535380 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5" pod "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" (UID: "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f"). InnerVolumeSpecName "kube-api-access-qsqd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:45.535497 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.535440 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca" pod "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" (UID: "8462d1df-2ed7-48fa-a7f6-dc4ae19e035f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:45.633925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.633857 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqd5\" (UniqueName: \"kubernetes.io/projected/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-kube-api-access-qsqd5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:45.633925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.633899 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:45.633925 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:45.633917 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.335341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:46.335310 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" Nov 01 22:37:46.335531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:46.335321 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv" event={"ID":"8462d1df-2ed7-48fa-a7f6-dc4ae19e035f","Type":"ContainerDied","Data":"21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd"} Nov 01 22:37:46.335531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:46.335426 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f80ed5fa953a50bfb9f4a31b737f35c6ecc3c83b7fb2c0207f6a497f86a6fd" Nov 01 22:37:46.369633 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:37:46.369608 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8462d1df_2ed7_48fa_a7f6_dc4ae19e035f.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:37:47.025898 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:47.025862 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt"] Nov 01 22:37:47.026313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:47.025910 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75fcb7c3-h8hkt"] Nov 01 22:37:47.163644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:47.163612 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4470a242-97cb-4f6d-914b-bc18924953f9" path="/var/lib/kubelet/pods/4470a242-97cb-4f6d-914b-bc18924953f9/volumes" Nov 01 22:37:52.304979 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:37:52.301990 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:37:52.419245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.419212 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp"] Nov 01 22:37:52.419607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.419595 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" containerName="revoke-keys" Nov 01 22:37:52.419652 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.419608 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" containerName="revoke-keys" Nov 01 22:37:52.419700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.419691 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" containerName="revoke-keys" Nov 01 22:37:52.423115 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.423100 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.432291 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.432266 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp"] Nov 01 22:37:52.445138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.445120 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:52.445263 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.445220 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:37:52.491566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.491542 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.491700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.491599 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.491773 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.491753 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgkf\" (UniqueName: \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.592569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.592494 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.592734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.592578 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgkf\" (UniqueName: \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.592734 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.592621 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.593290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.593272 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.594974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.594957 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.604388 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.604360 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgkf\" (UniqueName: \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\") pod \"maas-api-revoke-sub-5a29efb3-g64zp\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:52.738087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:52.738053 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:53.017165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:53.017143 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp"] Nov 01 22:37:53.018792 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:37:53.018766 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d83c4_9cec_41d5_8178_7cc0983882b4.slice/crio-0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9 WatchSource:0}: Error finding container 0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9: Status 404 returned error can't find the container with id 0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9 Nov 01 22:37:53.376769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:53.376682 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" event={"ID":"2d2d83c4-9cec-41d5-8178-7cc0983882b4","Type":"ContainerStarted","Data":"6770b6522af9d89dd992bdfc878942c9d452057310f924338c3ae549289566bb"} Nov 01 22:37:53.376769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:53.376751 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" event={"ID":"2d2d83c4-9cec-41d5-8178-7cc0983882b4","Type":"ContainerStarted","Data":"0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9"} Nov 01 22:37:53.396633 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:53.396575 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" podStartSLOduration=1.396558097 podStartE2EDuration="1.396558097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:53.395209547 +0000 UTC m=+3220.369112082" watchObservedRunningTime="2026-11-01 22:37:53.396558097 +0000 UTC m=+3220.370460637" Nov 01 22:37:54.387837 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:54.387794 4395 generic.go:356] "Generic (PLEG): container finished" podID="2d2d83c4-9cec-41d5-8178-7cc0983882b4" containerID="6770b6522af9d89dd992bdfc878942c9d452057310f924338c3ae549289566bb" exitCode=0 Nov 01 22:37:54.388370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:54.387887 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" event={"ID":"2d2d83c4-9cec-41d5-8178-7cc0983882b4","Type":"ContainerDied","Data":"6770b6522af9d89dd992bdfc878942c9d452057310f924338c3ae549289566bb"} Nov 01 22:37:56.178064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.175191 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:56.218563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.217305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\") pod \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " Nov 01 22:37:56.218563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.217357 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\" (UniqueName: \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\") pod \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " Nov 01 22:37:56.218563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.217431 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\") pod \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\" (UID: \"2d2d83c4-9cec-41d5-8178-7cc0983882b4\") " Nov 01 22:37:56.218563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.218250 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca" pod "2d2d83c4-9cec-41d5-8178-7cc0983882b4" (UID: "2d2d83c4-9cec-41d5-8178-7cc0983882b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:56.273465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.273012 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca" pod "2d2d83c4-9cec-41d5-8178-7cc0983882b4" (UID: "2d2d83c4-9cec-41d5-8178-7cc0983882b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:56.273465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.273151 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf" pod "2d2d83c4-9cec-41d5-8178-7cc0983882b4" (UID: "2d2d83c4-9cec-41d5-8178-7cc0983882b4"). InnerVolumeSpecName "kube-api-access-kkgkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:56.319002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.318957 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:56.319002 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.319003 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2d83c4-9cec-41d5-8178-7cc0983882b4-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:56.319365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.319020 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgkf\" (UniqueName: \"kubernetes.io/projected/2d2d83c4-9cec-41d5-8178-7cc0983882b4-kube-api-access-kkgkf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:56.449408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.449328 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" Nov 01 22:37:56.449547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.449327 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp" event={"ID":"2d2d83c4-9cec-41d5-8178-7cc0983882b4","Type":"ContainerDied","Data":"0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9"} Nov 01 22:37:56.449547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:37:56.449449 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0573e53e7ef39c96ad65d12ed612582cb01fcf6e93cffec1a819d92286cb5ee9" Nov 01 22:38:17.451429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.451399 4395 scope.go:122] "RemoveContainer" containerID="949a5753607ca236f0944c09354653a16038d43ace0cccc194296de0fdfa64bc" Nov 01 22:38:17.460740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.460720 4395 scope.go:122] "RemoveContainer" containerID="450cb827536e1a4991b7567a0ed87ade82ed1405dee3e75206578b3c3275b43c" Nov 01 22:38:17.469012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.468985 4395 scope.go:122] "RemoveContainer" containerID="c2b31227dedebf209f2e1bcfa4800e024d1604389a808b1ae287933cd287e2e0" Nov 01 22:38:17.477700 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.477683 4395 scope.go:122] "RemoveContainer" containerID="efcc54ac916594937cac8882cd5dd0d051fe76afb3bb5cc0730c554d48161960" Nov 01 22:38:17.486206 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.486192 4395 scope.go:122] "RemoveContainer" containerID="89c85ac4298d86e671f108a9e28c06060fd05657bda7a0f8f0ac6f59a1c2a79e" Nov 01 22:38:17.493882 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.493866 4395 scope.go:122] "RemoveContainer" containerID="43ceed1ee365aaa978ae0f07ab091e3262405490248922e62a54af7fa6a3bb6b" Nov 01 22:38:17.504040 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.504012 4395 scope.go:122] "RemoveContainer" containerID="88e958e163d6fe345e8d9ef4f488040df15d10aa2322363e086eb2fff5a23b3b" Nov 01 22:38:17.511891 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.511874 4395 scope.go:122] "RemoveContainer" containerID="6d003b20f7da29c516dd7dc2d03a89691a983ae5eea7c0bdc911ebc331a0b31d" Nov 01 22:38:17.519653 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:17.519637 4395 scope.go:122] "RemoveContainer" containerID="a592790c9f3ca44976674485c9fcbaa9e1b60d4fbd97cb1d768c706a5901197c" Nov 01 22:38:22.300766 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:38:22.300726 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:38:33.022642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:33.022608 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9"] Nov 01 22:38:33.024958 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:33.024933 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lkmf9"] Nov 01 22:38:33.163179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:33.163148 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f27c63b-520a-46b5-8680-c08721cbbe01" path="/var/lib/kubelet/pods/5f27c63b-520a-46b5-8680-c08721cbbe01/volumes" Nov 01 22:38:51.819348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.819312 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j"] Nov 01 22:38:51.823847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.823823 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zj44j"] Nov 01 22:38:51.843916 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.843890 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575"] Nov 01 22:38:51.844403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.844385 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2d2d83c4-9cec-41d5-8178-7cc0983882b4" containerName="revoke-keys" Nov 01 22:38:51.844458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.844407 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2d83c4-9cec-41d5-8178-7cc0983882b4" containerName="revoke-keys" Nov 01 22:38:51.844548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.844536 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2d83c4-9cec-41d5-8178-7cc0983882b4" containerName="revoke-keys" Nov 01 22:38:51.854451 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.854421 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:51.856559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.856532 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575"] Nov 01 22:38:51.858679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.858656 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:51.858875 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.858858 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:38:51.923445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.923410 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:51.923605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.923453 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:51.923605 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:51.923524 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcssf\" (UniqueName: \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.024376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.024342 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.024555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.024387 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.024555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.024500 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcssf\" (UniqueName: \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.024969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.024943 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.026957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.026927 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.037316 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.037295 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcssf\" (UniqueName: \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\") pod \"maas-api-revoke-sub-3d4156f1-cs575\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.169213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.169125 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:52.301002 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:38:52.300962 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:38:52.450066 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.450021 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575"] Nov 01 22:38:52.452388 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:38:52.452362 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756bcff1_8c20_4fd2_8443_5be85dcfb597.slice/crio-cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba WatchSource:0}: Error finding container cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba: Status 404 returned error can't find the container with id cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba Nov 01 22:38:52.756896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:52.756861 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" event={"ID":"756bcff1-8c20-4fd2-8443-5be85dcfb597","Type":"ContainerStarted","Data":"cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba"} Nov 01 22:38:53.165010 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:53.164944 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f40084-e5c0-4119-b2d0-abbba80c48c8" path="/var/lib/kubelet/pods/28f40084-e5c0-4119-b2d0-abbba80c48c8/volumes" Nov 01 22:38:53.764522 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:53.764482 4395 generic.go:356] "Generic (PLEG): container finished" podID="756bcff1-8c20-4fd2-8443-5be85dcfb597" containerID="1918c0ec14ed64dca5188ee8d098467889a2141b72f27dd1c27355e37273ee03" exitCode=0 Nov 01 22:38:53.764719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:53.764571 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" event={"ID":"756bcff1-8c20-4fd2-8443-5be85dcfb597","Type":"ContainerDied","Data":"1918c0ec14ed64dca5188ee8d098467889a2141b72f27dd1c27355e37273ee03"} Nov 01 22:38:54.906613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:54.906589 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:55.054491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.054406 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\") pod \"756bcff1-8c20-4fd2-8443-5be85dcfb597\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " Nov 01 22:38:55.054663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.054546 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\" (UniqueName: \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\") pod \"756bcff1-8c20-4fd2-8443-5be85dcfb597\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " Nov 01 22:38:55.054663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.054581 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\") pod \"756bcff1-8c20-4fd2-8443-5be85dcfb597\" (UID: \"756bcff1-8c20-4fd2-8443-5be85dcfb597\") " Nov 01 22:38:55.054990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.054966 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca" pod "756bcff1-8c20-4fd2-8443-5be85dcfb597" (UID: "756bcff1-8c20-4fd2-8443-5be85dcfb597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:55.056612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.056589 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca" pod "756bcff1-8c20-4fd2-8443-5be85dcfb597" (UID: "756bcff1-8c20-4fd2-8443-5be85dcfb597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:55.056676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.056651 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf" pod "756bcff1-8c20-4fd2-8443-5be85dcfb597" (UID: "756bcff1-8c20-4fd2-8443-5be85dcfb597"). InnerVolumeSpecName "kube-api-access-dcssf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:55.155573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.155540 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcssf\" (UniqueName: \"kubernetes.io/projected/756bcff1-8c20-4fd2-8443-5be85dcfb597-kube-api-access-dcssf\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:55.155573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.155568 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:55.155573 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.155578 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756bcff1-8c20-4fd2-8443-5be85dcfb597-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:55.776156 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.776122 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" Nov 01 22:38:55.776339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.776157 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575" event={"ID":"756bcff1-8c20-4fd2-8443-5be85dcfb597","Type":"ContainerDied","Data":"cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba"} Nov 01 22:38:55.776339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:38:55.776208 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbbe4a8f2325a2b62a55fc0bc805c05f8437247bb0df7ba5b400aaf0dcdf3dba" Nov 01 22:39:10.331140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.331100 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:39:10.331739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.331718 4395 cpu_manager.go:414] "containerMap: removing container" podUID="756bcff1-8c20-4fd2-8443-5be85dcfb597" containerName="revoke-keys" Nov 01 22:39:10.331832 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.331741 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756bcff1-8c20-4fd2-8443-5be85dcfb597" containerName="revoke-keys" Nov 01 22:39:10.331893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.331846 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="756bcff1-8c20-4fd2-8443-5be85dcfb597" containerName="revoke-keys" Nov 01 22:39:10.335250 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.335226 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.339063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.339042 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-096081-openshift-default-dockercfg-jwp4d\"" Nov 01 22:39:10.350795 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.350771 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:39:10.437507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437300 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.437507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437356 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.437507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437392 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.437507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437437 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.437507 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437468 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.438138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437560 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.438138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437589 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.438138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437693 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zrp\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.438138 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.437758 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539311 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zrp\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539383 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539420 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539454 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539484 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539527 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539632 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.539835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539660 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.540053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.539889 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.540112 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.540072 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.540307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.540219 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.540307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.540219 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.540307 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.540274 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.542077 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.542050 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.542215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.542191 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.556375 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.556356 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.558538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.558519 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zrp\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\") pod \"e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.649666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.649579 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:10.933199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.933170 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:39:10.936948 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:39:10.936915 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afd3bbd_e4b4_4866_a01a_f8d52f783479.slice/crio-6fae1726e2f57e5e3ec477c72f4a27b7251db736ca77e3e9f8678b6069b878ed WatchSource:0}: Error finding container 6fae1726e2f57e5e3ec477c72f4a27b7251db736ca77e3e9f8678b6069b878ed: Status 404 returned error can't find the container with id 6fae1726e2f57e5e3ec477c72f4a27b7251db736ca77e3e9f8678b6069b878ed Nov 01 22:39:10.939498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.939460 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:10.939614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.939539 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:10.939614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.939584 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:10.966065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:10.966013 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerStarted","Data":"6fae1726e2f57e5e3ec477c72f4a27b7251db736ca77e3e9f8678b6069b878ed"} Nov 01 22:39:11.972767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:11.972730 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerStarted","Data":"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d"} Nov 01 22:39:12.031874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:12.031817 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podStartSLOduration=2.031802708 podStartE2EDuration="2.031802708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:12.028781134 +0000 UTC m=+3299.002683675" watchObservedRunningTime="2026-11-01 22:39:12.031802708 +0000 UTC m=+3299.005705246" Nov 01 22:39:12.650531 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:12.650496 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:12.655495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:12.655471 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:12.977733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:12.977651 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:12.978666 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:12.978649 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:15.491518 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.491480 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:39:15.495272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.495245 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.505089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.505064 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-096081\"" Nov 01 22:39:15.506299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.506270 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fc5cbdbf-dockercfg-pwlkj\"" Nov 01 22:39:15.509721 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.509699 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:39:15.587932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.587900 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.588160 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.587953 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8px4r\" (UniqueName: \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.689550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.689514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.689730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.689559 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8px4r\" (UniqueName: \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.690289 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.690265 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.702433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.702403 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8px4r\" (UniqueName: \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\") pod \"payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:15.808548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:15.808525 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:16.087353 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:16.087326 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:39:16.089465 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:39:16.089434 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e52b5f_edd7_4811_97e1_e5fb80abbcbd.slice/crio-8a9bea790cba2e4f9044a2f12bcc0b90fb04ec98e5e491b789d737aabeb820b8 WatchSource:0}: Error finding container 8a9bea790cba2e4f9044a2f12bcc0b90fb04ec98e5e491b789d737aabeb820b8: Status 404 returned error can't find the container with id 8a9bea790cba2e4f9044a2f12bcc0b90fb04ec98e5e491b789d737aabeb820b8 Nov 01 22:39:17.004864 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.004815 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" event={"ID":"31e52b5f-edd7-4811-97e1-e5fb80abbcbd","Type":"ContainerStarted","Data":"e7e0369fab71cc9c6f871df3d75b6de2eccadccf6a4aa18530e3072bc30ae353"} Nov 01 22:39:17.005288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.004954 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:17.005288 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.004973 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" event={"ID":"31e52b5f-edd7-4811-97e1-e5fb80abbcbd","Type":"ContainerStarted","Data":"8a9bea790cba2e4f9044a2f12bcc0b90fb04ec98e5e491b789d737aabeb820b8"} Nov 01 22:39:17.022800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.022751 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" podStartSLOduration=2.022728952 podStartE2EDuration="2.022728952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:17.02241574 +0000 UTC m=+3303.996318279" watchObservedRunningTime="2026-11-01 22:39:17.022728952 +0000 UTC m=+3303.996631490" Nov 01 22:39:17.583595 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.583555 4395 scope.go:122] "RemoveContainer" containerID="199c89f753d97f7f9fe88bea3a33e68339ddace0d18c2e983842551e15d1be48" Nov 01 22:39:17.598199 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:17.598007 4395 scope.go:122] "RemoveContainer" containerID="5d58f171f6dec4f36370b160d9182cd0419dcb26d4417eb4a837449b877584ff" Nov 01 22:39:22.325355 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:39:22.325313 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:39:27.979514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:27.979410 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:27.979514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:27.979476 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:28.013757 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.013720 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:39:28.024060 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.024013 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm"] Nov 01 22:39:28.026241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.026221 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-sdspm"] Nov 01 22:39:28.069212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.069168 4395 generic.go:356] "Generic (PLEG): container finished" podID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerID="3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d" exitCode=0 Nov 01 22:39:28.069212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.069212 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerDied","Data":"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d"} Nov 01 22:39:28.069668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.069646 4395 scope.go:122] "RemoveContainer" containerID="3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d" Nov 01 22:39:28.071090 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.071055 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:28.071175 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.071152 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:28.071224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:28.071199 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:39:29.101453 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:29.101414 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerStarted","Data":"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53"} Nov 01 22:39:29.190054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:29.174956 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74102a24-7d48-4bb4-8e65-53ae0d45b6a1" path="/var/lib/kubelet/pods/74102a24-7d48-4bb4-8e65-53ae0d45b6a1/volumes" Nov 01 22:39:29.650366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:29.650329 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:29.650560 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:29.650531 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:29.650614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:29.650581 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:30.650442 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:30.650409 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:30.650974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:30.650464 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:31.650913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:31.650876 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:31.651348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:31.650935 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:32.650186 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:32.650149 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:32.650380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:32.650203 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:33.649947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:33.649915 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Nov 01 22:39:33.650435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:33.649966 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Nov 01 22:39:34.654530 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:34.654494 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:34.654937 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:34.654821 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:34.655584 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:34.655567 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:39:47.035124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:47.035091 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4"] Nov 01 22:39:47.039216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:47.039185 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m9xg4"] Nov 01 22:39:47.163718 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:47.163686 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e20e20c-a016-430c-a065-d44acb05e5db" path="/var/lib/kubelet/pods/2e20e20c-a016-430c-a065-d44acb05e5db/volumes" Nov 01 22:39:52.302055 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:39:52.302002 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:39:58.021851 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:58.021816 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd"] Nov 01 22:39:58.025729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:58.025705 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2d5wd"] Nov 01 22:39:59.164383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:39:59.164353 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a8b995-480a-4105-9d7d-25c07c8a8522" path="/var/lib/kubelet/pods/94a8b995-480a-4105-9d7d-25c07c8a8522/volumes" Nov 01 22:40:02.992502 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:02.992468 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9"] Nov 01 22:40:02.997290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:02.997274 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.002800 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.002755 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:03.003007 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.002980 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:40:03.005838 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.005743 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9"] Nov 01 22:40:03.042692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.042659 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.042818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.042771 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.042818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.042816 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq29\" (UniqueName: \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.143786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.143758 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.143895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.143800 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.143895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.143826 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq29\" (UniqueName: \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.144433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.144404 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.146278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.146253 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.156418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.156396 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq29\" (UniqueName: \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\") pod \"maas-api-revoke-sub-b61e9352-w6dm9\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.312736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.312707 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:03.588359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:03.588279 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9"] Nov 01 22:40:03.591284 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:40:03.591255 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b26a34_e6c1_4d4a_81a1_7e2764a023b2.slice/crio-b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce WatchSource:0}: Error finding container b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce: Status 404 returned error can't find the container with id b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce Nov 01 22:40:04.297180 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:04.297144 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" event={"ID":"93b26a34-e6c1-4d4a-81a1-7e2764a023b2","Type":"ContainerStarted","Data":"17948908f30ffb5c33ebf1be68789a059546a72607d87898447c16f2f1fcf1b3"} Nov 01 22:40:04.297596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:04.297206 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" event={"ID":"93b26a34-e6c1-4d4a-81a1-7e2764a023b2","Type":"ContainerStarted","Data":"b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce"} Nov 01 22:40:04.316868 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:04.316816 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" podStartSLOduration=2.316802851 podStartE2EDuration="2.316802851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:04.31595971 +0000 UTC m=+3351.289862249" watchObservedRunningTime="2026-11-01 22:40:04.316802851 +0000 UTC m=+3351.290705414" Nov 01 22:40:05.303924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:05.303889 4395 generic.go:356] "Generic (PLEG): container finished" podID="93b26a34-e6c1-4d4a-81a1-7e2764a023b2" containerID="17948908f30ffb5c33ebf1be68789a059546a72607d87898447c16f2f1fcf1b3" exitCode=0 Nov 01 22:40:05.304365 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:05.303983 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" event={"ID":"93b26a34-e6c1-4d4a-81a1-7e2764a023b2","Type":"ContainerDied","Data":"17948908f30ffb5c33ebf1be68789a059546a72607d87898447c16f2f1fcf1b3"} Nov 01 22:40:06.441621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.441599 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:06.493753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.493715 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\") pod \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " Nov 01 22:40:06.493949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.493796 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\" (UniqueName: \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\") pod \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " Nov 01 22:40:06.493949 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.493853 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\") pod \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\" (UID: \"93b26a34-e6c1-4d4a-81a1-7e2764a023b2\") " Nov 01 22:40:06.494127 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.494100 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca" pod "93b26a34-e6c1-4d4a-81a1-7e2764a023b2" (UID: "93b26a34-e6c1-4d4a-81a1-7e2764a023b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:06.494339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.494319 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:06.496106 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.496083 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29" pod "93b26a34-e6c1-4d4a-81a1-7e2764a023b2" (UID: "93b26a34-e6c1-4d4a-81a1-7e2764a023b2"). InnerVolumeSpecName "kube-api-access-bwq29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:06.496547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.496529 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca" pod "93b26a34-e6c1-4d4a-81a1-7e2764a023b2" (UID: "93b26a34-e6c1-4d4a-81a1-7e2764a023b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:06.595401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.595334 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq29\" (UniqueName: \"kubernetes.io/projected/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-kube-api-access-bwq29\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:06.595401 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:06.595359 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93b26a34-e6c1-4d4a-81a1-7e2764a023b2-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:07.316541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:07.316506 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" event={"ID":"93b26a34-e6c1-4d4a-81a1-7e2764a023b2","Type":"ContainerDied","Data":"b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce"} Nov 01 22:40:07.316541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:07.316524 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9" Nov 01 22:40:07.316730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:07.316562 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5eee63171839f9776d4bbc3c2dfcac33e5dbbd0e5d86aa2d858e60b28770cce" Nov 01 22:40:14.248969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.248928 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z"] Nov 01 22:40:14.249380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.249345 4395 cpu_manager.go:414] "containerMap: removing container" podUID="93b26a34-e6c1-4d4a-81a1-7e2764a023b2" containerName="revoke-keys" Nov 01 22:40:14.249380 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.249357 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b26a34-e6c1-4d4a-81a1-7e2764a023b2" containerName="revoke-keys" Nov 01 22:40:14.249473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.249461 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b26a34-e6c1-4d4a-81a1-7e2764a023b2" containerName="revoke-keys" Nov 01 22:40:14.252676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.252661 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.261969 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.261948 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:14.262120 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.262050 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:40:14.271319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.271297 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z"] Nov 01 22:40:14.371730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.371700 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.371919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.371736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.371919 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.371778 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bnq\" (UniqueName: \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.472955 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.472926 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.473170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.472964 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.473170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.472993 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bnq\" (UniqueName: \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.473714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.473693 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.475514 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.475490 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.500129 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.500069 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bnq\" (UniqueName: \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.565195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.565169 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:14.853984 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:14.853953 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z"] Nov 01 22:40:14.855160 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:40:14.855130 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ae4d79_c582_438c_a60d_ea3eb81e277a.slice/crio-ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e WatchSource:0}: Error finding container ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e: Status 404 returned error can't find the container with id ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e Nov 01 22:40:15.361309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:15.361270 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" event={"ID":"f3ae4d79-c582-438c-a60d-ea3eb81e277a","Type":"ContainerStarted","Data":"210834bf6a7ada7f85ededa59abb47a8e07c3848bd8c1685cdcb64238db2d67e"} Nov 01 22:40:15.361675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:15.361342 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" event={"ID":"f3ae4d79-c582-438c-a60d-ea3eb81e277a","Type":"ContainerStarted","Data":"ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e"} Nov 01 22:40:15.390403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:15.390351 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" podStartSLOduration=1.390337259 podStartE2EDuration="1.390337259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:15.388713572 +0000 UTC m=+3362.362616111" watchObservedRunningTime="2026-11-01 22:40:15.390337259 +0000 UTC m=+3362.364239797" Nov 01 22:40:16.368704 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:16.368666 4395 generic.go:356] "Generic (PLEG): container finished" podID="f3ae4d79-c582-438c-a60d-ea3eb81e277a" containerID="210834bf6a7ada7f85ededa59abb47a8e07c3848bd8c1685cdcb64238db2d67e" exitCode=0 Nov 01 22:40:16.369174 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:16.368749 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" event={"ID":"f3ae4d79-c582-438c-a60d-ea3eb81e277a","Type":"ContainerDied","Data":"210834bf6a7ada7f85ededa59abb47a8e07c3848bd8c1685cdcb64238db2d67e"} Nov 01 22:40:17.537506 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.537477 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:17.615871 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.615844 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\") pod \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " Nov 01 22:40:17.616042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.615932 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\") pod \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " Nov 01 22:40:17.616042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.615971 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\" (UniqueName: \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\") pod \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\" (UID: \"f3ae4d79-c582-438c-a60d-ea3eb81e277a\") " Nov 01 22:40:17.616313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.616286 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca" pod "f3ae4d79-c582-438c-a60d-ea3eb81e277a" (UID: "f3ae4d79-c582-438c-a60d-ea3eb81e277a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:17.616427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.616412 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:17.618167 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.618147 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq" pod "f3ae4d79-c582-438c-a60d-ea3eb81e277a" (UID: "f3ae4d79-c582-438c-a60d-ea3eb81e277a"). InnerVolumeSpecName "kube-api-access-c8bnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:17.618249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.618206 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca" pod "f3ae4d79-c582-438c-a60d-ea3eb81e277a" (UID: "f3ae4d79-c582-438c-a60d-ea3eb81e277a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:17.623424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.623403 4395 scope.go:122] "RemoveContainer" containerID="e278ed64a3dd3abd5d101b1d12f8f536954ca2c9575a028a8cc49ffe81a5e6f5" Nov 01 22:40:17.717415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.717384 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bnq\" (UniqueName: \"kubernetes.io/projected/f3ae4d79-c582-438c-a60d-ea3eb81e277a-kube-api-access-c8bnq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:17.717415 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:17.717408 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ae4d79-c582-438c-a60d-ea3eb81e277a-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:18.380998 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:18.380969 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" Nov 01 22:40:18.381196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:18.380999 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z" event={"ID":"f3ae4d79-c582-438c-a60d-ea3eb81e277a","Type":"ContainerDied","Data":"ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e"} Nov 01 22:40:18.381196 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:18.381055 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9753d7f4f9d35f410fa837ad4903bf15ee55ab4d6651407b2e10383751308e" Nov 01 22:40:22.300775 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:40:22.300735 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:40:24.027712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:24.027669 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl"] Nov 01 22:40:24.030101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:24.030084 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-wsmzl"] Nov 01 22:40:24.370806 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:24.370688 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:40:25.164262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:25.164223 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4284d1c-ee57-4f6f-bb3a-5019a50ad46a" path="/var/lib/kubelet/pods/d4284d1c-ee57-4f6f-bb3a-5019a50ad46a/volumes" Nov 01 22:40:25.313927 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:25.313898 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z"] Nov 01 22:40:25.316644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:25.316617 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-096081-jrf6z"] Nov 01 22:40:27.163846 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:27.163816 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ae4d79-c582-438c-a60d-ea3eb81e277a" path="/var/lib/kubelet/pods/f3ae4d79-c582-438c-a60d-ea3eb81e277a/volumes" Nov 01 22:40:29.371644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:29.371585 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" containerID="cri-o://e7e0369fab71cc9c6f871df3d75b6de2eccadccf6a4aa18530e3072bc30ae353" gracePeriod=25 Nov 01 22:40:32.123071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:32.123012 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:40:32.123655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:32.123371 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" containerID="cri-o://48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53" gracePeriod=30 Nov 01 22:40:33.314753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.314717 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575"] Nov 01 22:40:33.320550 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.320523 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cs575"] Nov 01 22:40:33.341824 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.341799 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn"] Nov 01 22:40:33.342220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.342207 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f3ae4d79-c582-438c-a60d-ea3eb81e277a" containerName="revoke-keys" Nov 01 22:40:33.342272 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.342222 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ae4d79-c582-438c-a60d-ea3eb81e277a" containerName="revoke-keys" Nov 01 22:40:33.342356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.342306 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ae4d79-c582-438c-a60d-ea3eb81e277a" containerName="revoke-keys" Nov 01 22:40:33.346680 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.346663 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.352619 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.352593 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:33.353065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.353016 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:40:33.355567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.355544 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn"] Nov 01 22:40:33.380551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.380529 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5g7\" (UniqueName: \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.380673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.380561 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.380673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.380580 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.481293 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.481269 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5g7\" (UniqueName: \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.481438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.481306 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.481438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.481325 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.481934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.481915 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.483840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.483820 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.493730 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.493709 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5g7\" (UniqueName: \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\") pod \"maas-api-revoke-sub-3d4156f1-fdxzn\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.660677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.660614 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:33.939959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:33.939933 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn"] Nov 01 22:40:33.941890 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:40:33.941864 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18b196b_daee_4a2b_ab69_4da3d0f0e828.slice/crio-afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4 WatchSource:0}: Error finding container afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4: Status 404 returned error can't find the container with id afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4 Nov 01 22:40:34.020847 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.020823 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8"] Nov 01 22:40:34.024011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.023986 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-7xtl8"] Nov 01 22:40:34.471197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.471165 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" event={"ID":"c18b196b-daee-4a2b-ab69-4da3d0f0e828","Type":"ContainerStarted","Data":"caf17897cabd673cdd3aeeed1934565d2c0f65ea2fef60026da9b0a88260fc1d"} Nov 01 22:40:34.471589 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.471214 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" event={"ID":"c18b196b-daee-4a2b-ab69-4da3d0f0e828","Type":"ContainerStarted","Data":"afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4"} Nov 01 22:40:34.492014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.491957 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" podStartSLOduration=1.491938764 podStartE2EDuration="1.491938764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:34.489015331 +0000 UTC m=+3381.462917871" watchObservedRunningTime="2026-11-01 22:40:34.491938764 +0000 UTC m=+3381.465841303" Nov 01 22:40:34.657508 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.657475 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:40:34.657649 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:34.657525 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:40:35.163713 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:35.163682 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f33839-441f-449d-b10c-adc4218aa351" path="/var/lib/kubelet/pods/45f33839-441f-449d-b10c-adc4218aa351/volumes" Nov 01 22:40:35.164125 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:35.164109 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756bcff1-8c20-4fd2-8443-5be85dcfb597" path="/var/lib/kubelet/pods/756bcff1-8c20-4fd2-8443-5be85dcfb597/volumes" Nov 01 22:40:35.477644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:35.477568 4395 generic.go:356] "Generic (PLEG): container finished" podID="c18b196b-daee-4a2b-ab69-4da3d0f0e828" containerID="caf17897cabd673cdd3aeeed1934565d2c0f65ea2fef60026da9b0a88260fc1d" exitCode=0 Nov 01 22:40:35.477644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:35.477618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" event={"ID":"c18b196b-daee-4a2b-ab69-4da3d0f0e828","Type":"ContainerDied","Data":"caf17897cabd673cdd3aeeed1934565d2c0f65ea2fef60026da9b0a88260fc1d"} Nov 01 22:40:36.621855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.621829 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:36.712165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.712137 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\" (UniqueName: \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\") pod \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " Nov 01 22:40:36.712362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.712221 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\") pod \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " Nov 01 22:40:36.712362 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.712248 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\") pod \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\" (UID: \"c18b196b-daee-4a2b-ab69-4da3d0f0e828\") " Nov 01 22:40:36.712626 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.712603 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca" pod "c18b196b-daee-4a2b-ab69-4da3d0f0e828" (UID: "c18b196b-daee-4a2b-ab69-4da3d0f0e828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:36.714537 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.714509 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca" pod "c18b196b-daee-4a2b-ab69-4da3d0f0e828" (UID: "c18b196b-daee-4a2b-ab69-4da3d0f0e828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:36.714612 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.714517 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7" pod "c18b196b-daee-4a2b-ab69-4da3d0f0e828" (UID: "c18b196b-daee-4a2b-ab69-4da3d0f0e828"). InnerVolumeSpecName "kube-api-access-mg5g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:36.813658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.813628 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:36.813658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.813655 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18b196b-daee-4a2b-ab69-4da3d0f0e828-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:36.813658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:36.813664 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5g7\" (UniqueName: \"kubernetes.io/projected/c18b196b-daee-4a2b-ab69-4da3d0f0e828-kube-api-access-mg5g7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.372993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.372968 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:40:37.418805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.418731 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.418805 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.418779 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.418994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.418819 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.418994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.418931 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.418995 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419050 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419109 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419145 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419179 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\") pod \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\" (UID: \"3afd3bbd-e4b4-4866-a01a-f8d52f783479\") " Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419208 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419251 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419334 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419445 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:37.419590 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419475 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:37.419920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419650 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.419920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419672 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.419920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419686 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.419920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419702 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.419920 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.419716 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.421836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.421809 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:40:37.421921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.421840 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "kube-api-access-l9zrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:37.421921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.421888 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:40:37.422006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.421922 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token" pod "3afd3bbd-e4b4-4866-a01a-f8d52f783479" (UID: "3afd3bbd-e4b4-4866-a01a-f8d52f783479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:37.489369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.489343 4395 generic.go:356] "Generic (PLEG): container finished" podID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerID="48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53" exitCode=0 Nov 01 22:40:37.489515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.489394 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" Nov 01 22:40:37.489515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.489424 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerDied","Data":"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53"} Nov 01 22:40:37.489515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.489482 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz" event={"ID":"3afd3bbd-e4b4-4866-a01a-f8d52f783479","Type":"ContainerDied","Data":"6fae1726e2f57e5e3ec477c72f4a27b7251db736ca77e3e9f8678b6069b878ed"} Nov 01 22:40:37.489515 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.489511 4395 scope.go:122] "RemoveContainer" containerID="48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53" Nov 01 22:40:37.493329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.493266 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" event={"ID":"c18b196b-daee-4a2b-ab69-4da3d0f0e828","Type":"ContainerDied","Data":"afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4"} Nov 01 22:40:37.493329 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.493309 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdb6e67a8e920d570f3e51516792f7bcea935ef7d9ea7472e728ceb37794ab4" Nov 01 22:40:37.493495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.493353 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn" Nov 01 22:40:37.499273 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.499256 4395 scope.go:122] "RemoveContainer" containerID="3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d" Nov 01 22:40:37.507245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.507227 4395 scope.go:122] "RemoveContainer" containerID="48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53" Nov 01 22:40:37.507486 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:40:37.507469 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53\": container with ID starting with 48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53 not found: ID does not exist" containerID="48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53" Nov 01 22:40:37.507563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.507497 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53"} err="failed to get container status \"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53\": rpc error: code = NotFound desc = could not find container \"48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53\": container with ID starting with 48f0e270ee5f1732a4b479d97287edaf476dba3cfccb2daf381eeede33361c53 not found: ID does not exist" Nov 01 22:40:37.507563 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.507520 4395 scope.go:122] "RemoveContainer" containerID="3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d" Nov 01 22:40:37.507754 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:40:37.507735 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d\": container with ID starting with 3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d not found: ID does not exist" containerID="3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d" Nov 01 22:40:37.507808 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.507759 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d"} err="failed to get container status \"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d\": rpc error: code = NotFound desc = could not find container \"3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d\": container with ID starting with 3afe99b0c6f09aa52f50028fc162d484e4fdccfa76703624703fbec736548f9d not found: ID does not exist" Nov 01 22:40:37.514860 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.514841 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:40:37.520248 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.520226 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-096081-openshift-default-5bf85f6d49s7btz"] Nov 01 22:40:37.520339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.520292 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zrp\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-kube-api-access-l9zrp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.520339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.520310 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.520339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.520320 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.520339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:37.520329 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd3bbd-e4b4-4866-a01a-f8d52f783479-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:38.012331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:38.012289 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:40:39.163094 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:39.163064 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" path="/var/lib/kubelet/pods/3afd3bbd-e4b4-4866-a01a-f8d52f783479/volumes" Nov 01 22:40:46.909169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:46.909134 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:46.909774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:46.909732 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:47.266867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:47.266828 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:47.267226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:47.267195 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.012342 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:48.012306 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:40:49.593422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.593227 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerID="e7e0369fab71cc9c6f871df3d75b6de2eccadccf6a4aa18530e3072bc30ae353" exitCode=0 Nov 01 22:40:49.593422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.593304 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" event={"ID":"31e52b5f-edd7-4811-97e1-e5fb80abbcbd","Type":"ContainerDied","Data":"e7e0369fab71cc9c6f871df3d75b6de2eccadccf6a4aa18530e3072bc30ae353"} Nov 01 22:40:49.625586 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.625559 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:40:49.682266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.682191 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\") pod \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " Nov 01 22:40:49.682409 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.682301 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\" (UniqueName: \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\") pod \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\" (UID: \"31e52b5f-edd7-4811-97e1-e5fb80abbcbd\") " Nov 01 22:40:49.682640 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.682615 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume" pod "31e52b5f-edd7-4811-97e1-e5fb80abbcbd" (UID: "31e52b5f-edd7-4811-97e1-e5fb80abbcbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:49.684548 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.684532 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r" pod "31e52b5f-edd7-4811-97e1-e5fb80abbcbd" (UID: "31e52b5f-edd7-4811-97e1-e5fb80abbcbd"). InnerVolumeSpecName "kube-api-access-8px4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:49.783648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.783628 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8px4r\" (UniqueName: \"kubernetes.io/projected/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-kube-api-access-8px4r\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:49.783648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:49.783650 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e52b5f-edd7-4811-97e1-e5fb80abbcbd-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:50.600439 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:50.600407 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" Nov 01 22:40:50.600913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:50.600444 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr" event={"ID":"31e52b5f-edd7-4811-97e1-e5fb80abbcbd","Type":"ContainerDied","Data":"8a9bea790cba2e4f9044a2f12bcc0b90fb04ec98e5e491b789d737aabeb820b8"} Nov 01 22:40:50.600913 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:50.600506 4395 scope.go:122] "RemoveContainer" containerID="e7e0369fab71cc9c6f871df3d75b6de2eccadccf6a4aa18530e3072bc30ae353" Nov 01 22:40:50.623691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:50.623664 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:40:50.628371 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:50.628345 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-096081-7d87c2dmwr"] Nov 01 22:40:51.163705 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:51.163664 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" path="/var/lib/kubelet/pods/31e52b5f-edd7-4811-97e1-e5fb80abbcbd/volumes" Nov 01 22:40:52.300785 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:40:52.300749 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:40:54.021747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:54.021711 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv"] Nov 01 22:40:54.025600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:54.025574 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-c7hpv"] Nov 01 22:40:55.163679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:40:55.163646 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ff2446-c6ad-4734-86ff-af58f420a517" path="/var/lib/kubelet/pods/98ff2446-c6ad-4734-86ff-af58f420a517/volumes" Nov 01 22:41:12.022155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:12.022112 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq"] Nov 01 22:41:12.024101 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:12.024078 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-jghjq"] Nov 01 22:41:13.163261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:13.163229 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da32023e-2310-47e6-90f9-76e97e0431a4" path="/var/lib/kubelet/pods/da32023e-2310-47e6-90f9-76e97e0431a4/volumes" Nov 01 22:41:17.646348 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:17.646318 4395 scope.go:122] "RemoveContainer" containerID="1846ea622d0e0882c200a70d6dd4853deb00ed028009b95d623a95f334b7173f" Nov 01 22:41:17.654744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:17.654723 4395 scope.go:122] "RemoveContainer" containerID="eea6bf865c33c49d46761d095fe3726502ac3706197a9672437f3d257ff1272d" Nov 01 22:41:17.662588 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:17.662571 4395 scope.go:122] "RemoveContainer" containerID="73edcfca1acf79f94108a1e18d9a56d95a4b45c75c38a7f3c55b360846901f9f" Nov 01 22:41:22.300841 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:41:22.300807 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:41:31.024164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:31.024128 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6"] Nov 01 22:41:31.027312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:31.027284 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wtfr6"] Nov 01 22:41:31.163386 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:31.163357 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48bca65-63ba-44e4-bdd0-2346849a7193" path="/var/lib/kubelet/pods/b48bca65-63ba-44e4-bdd0-2346849a7193/volumes" Nov 01 22:41:49.022744 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:49.022709 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5"] Nov 01 22:41:49.024578 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:49.024551 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vvbv5"] Nov 01 22:41:49.163574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:41:49.163538 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b42e10e-0319-46f1-9835-ce0bb45211c8" path="/var/lib/kubelet/pods/5b42e10e-0319-46f1-9835-ce0bb45211c8/volumes" Nov 01 22:41:52.302478 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:41:52.302433 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:42:17.462987 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.462952 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc"] Nov 01 22:42:17.463478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463436 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463452 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463464 4395 cpu_manager.go:414] "containerMap: removing container" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463472 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463497 4395 cpu_manager.go:414] "containerMap: removing container" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463504 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463511 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c18b196b-daee-4a2b-ab69-4da3d0f0e828" containerName="revoke-keys" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463516 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18b196b-daee-4a2b-ab69-4da3d0f0e828" containerName="revoke-keys" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463578 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463585 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afd3bbd-e4b4-4866-a01a-f8d52f783479" containerName="istio-proxy" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463595 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e52b5f-edd7-4811-97e1-e5fb80abbcbd" containerName="payload-pre-processing" Nov 01 22:42:17.463627 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.463602 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18b196b-daee-4a2b-ab69-4da3d0f0e828" containerName="revoke-keys" Nov 01 22:42:17.466643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.466623 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.473667 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.472930 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:42:17.474085 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.474041 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:42:17.476707 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.476684 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc"] Nov 01 22:42:17.592023 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.591991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.592168 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.592053 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.592216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.592181 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trqhp\" (UniqueName: \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.692729 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.692706 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.692870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.692743 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.692870 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.692798 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqhp\" (UniqueName: \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.693383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.693358 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.695152 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.695137 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.698593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.698573 4395 scope.go:122] "RemoveContainer" containerID="15d8cefc04bd8b308dd3a5c9560e1970e3e9ff635c244613e9b6e3864cb3c14f" Nov 01 22:42:17.705516 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.705496 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqhp\" (UniqueName: \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\") pod \"maas-api-revoke-sub-d576e68b-xldbc\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:17.709924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.709905 4395 scope.go:122] "RemoveContainer" containerID="0de0da16748adcd34e97e67584e8b3e5ae8e0cd97f2646e607ca8d36577f148d" Nov 01 22:42:17.718735 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.718712 4395 scope.go:122] "RemoveContainer" containerID="6cfc54f819a389bf7224c6911d90a4806583830db3a750c84bb6d998d48da3bd" Nov 01 22:42:17.726061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.726045 4395 scope.go:122] "RemoveContainer" containerID="c8ff2cd7db9bccd914f718367d7cea03af65adb0cc6cfccbc98774acae0735f0" Nov 01 22:42:17.781381 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:17.781360 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:18.058822 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:18.058799 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc"] Nov 01 22:42:18.061506 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:42:18.061475 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81d289e_f600_4317_bfc5_a97742ef67b8.slice/crio-002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c WatchSource:0}: Error finding container 002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c: Status 404 returned error can't find the container with id 002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c Nov 01 22:42:18.076239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:18.076211 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" event={"ID":"a81d289e-f600-4317-bfc5-a97742ef67b8","Type":"ContainerStarted","Data":"002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c"} Nov 01 22:42:19.084988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:19.084956 4395 generic.go:356] "Generic (PLEG): container finished" podID="a81d289e-f600-4317-bfc5-a97742ef67b8" containerID="f35dc2a538211fc522b17f31cf88803c3dbe088c9932bd381789fcecc87debf8" exitCode=0 Nov 01 22:42:19.085404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:19.085045 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" event={"ID":"a81d289e-f600-4317-bfc5-a97742ef67b8","Type":"ContainerDied","Data":"f35dc2a538211fc522b17f31cf88803c3dbe088c9932bd381789fcecc87debf8"} Nov 01 22:42:20.227105 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.227082 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:20.316411 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.316377 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\") pod \"a81d289e-f600-4317-bfc5-a97742ef67b8\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " Nov 01 22:42:20.316575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.316507 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\") pod \"a81d289e-f600-4317-bfc5-a97742ef67b8\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " Nov 01 22:42:20.316660 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.316580 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\" (UniqueName: \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\") pod \"a81d289e-f600-4317-bfc5-a97742ef67b8\" (UID: \"a81d289e-f600-4317-bfc5-a97742ef67b8\") " Nov 01 22:42:20.316872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.316850 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca" pod "a81d289e-f600-4317-bfc5-a97742ef67b8" (UID: "a81d289e-f600-4317-bfc5-a97742ef67b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:20.318542 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.318521 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca" pod "a81d289e-f600-4317-bfc5-a97742ef67b8" (UID: "a81d289e-f600-4317-bfc5-a97742ef67b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:20.318668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.318649 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp" pod "a81d289e-f600-4317-bfc5-a97742ef67b8" (UID: "a81d289e-f600-4317-bfc5-a97742ef67b8"). InnerVolumeSpecName "kube-api-access-trqhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:20.417489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.417419 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:20.417489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.417446 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trqhp\" (UniqueName: \"kubernetes.io/projected/a81d289e-f600-4317-bfc5-a97742ef67b8-kube-api-access-trqhp\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:20.417489 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:20.417456 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81d289e-f600-4317-bfc5-a97742ef67b8-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:21.102970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:21.102940 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" Nov 01 22:42:21.102970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:21.102953 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc" event={"ID":"a81d289e-f600-4317-bfc5-a97742ef67b8","Type":"ContainerDied","Data":"002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c"} Nov 01 22:42:21.103200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:21.103022 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="002f4bf284708216c1ab415dcde4debad41bb4efc3e3debc0a9fd795265d7b2c" Nov 01 22:42:22.300597 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:42:22.300563 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:42:36.176208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.176173 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt"] Nov 01 22:42:36.176637 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.176622 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a81d289e-f600-4317-bfc5-a97742ef67b8" containerName="revoke-keys" Nov 01 22:42:36.176683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.176638 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81d289e-f600-4317-bfc5-a97742ef67b8" containerName="revoke-keys" Nov 01 22:42:36.176737 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.176726 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81d289e-f600-4317-bfc5-a97742ef67b8" containerName="revoke-keys" Nov 01 22:42:36.180197 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.180171 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.193902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.191256 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:42:36.193902 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.191342 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:42:36.194208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.194184 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt"] Nov 01 22:42:36.295728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.295701 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.295877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.295754 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.295877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.295857 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4x8\" (UniqueName: \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.396914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.396888 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4x8\" (UniqueName: \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.397083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.396947 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.397083 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.396990 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.397672 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.397647 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.399448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.399427 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.409172 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.409152 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4x8\" (UniqueName: \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\") pod \"maas-api-revoke-sub-d2bc93bc-k2wpt\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.500689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.500669 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:36.780858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:36.780747 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt"] Nov 01 22:42:36.783177 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:42:36.783149 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bef7ad0_3487_4e4c_a64e_7762eb4bdef4.slice/crio-06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3 WatchSource:0}: Error finding container 06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3: Status 404 returned error can't find the container with id 06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3 Nov 01 22:42:37.194213 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:37.194178 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" event={"ID":"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4","Type":"ContainerStarted","Data":"d9f30d3b6a3bf99a32e4d790fe43687e486725ed95ded113912caa80bf4e6636"} Nov 01 22:42:37.194854 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:37.194243 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" event={"ID":"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4","Type":"ContainerStarted","Data":"06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3"} Nov 01 22:42:37.213093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:37.213020 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" podStartSLOduration=1.213009606 podStartE2EDuration="1.213009606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:37.212152158 +0000 UTC m=+3504.186054698" watchObservedRunningTime="2026-11-01 22:42:37.213009606 +0000 UTC m=+3504.186912145" Nov 01 22:42:38.200786 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:38.200752 4395 generic.go:356] "Generic (PLEG): container finished" podID="6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" containerID="d9f30d3b6a3bf99a32e4d790fe43687e486725ed95ded113912caa80bf4e6636" exitCode=0 Nov 01 22:42:38.201258 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:38.200830 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" event={"ID":"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4","Type":"ContainerDied","Data":"d9f30d3b6a3bf99a32e4d790fe43687e486725ed95ded113912caa80bf4e6636"} Nov 01 22:42:39.344374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.344348 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:39.422339 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.422309 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\" (UniqueName: \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\") pod \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " Nov 01 22:42:39.422505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.422367 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\") pod \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " Nov 01 22:42:39.422505 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.422471 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\") pod \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\" (UID: \"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4\") " Nov 01 22:42:39.422804 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.422781 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca" pod "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" (UID: "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:39.424574 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.424550 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8" pod "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" (UID: "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4"). InnerVolumeSpecName "kube-api-access-6z4x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:39.424683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.424598 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca" pod "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" (UID: "6bef7ad0-3487-4e4c-a64e-7762eb4bdef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:39.523369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.523347 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4x8\" (UniqueName: \"kubernetes.io/projected/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-kube-api-access-6z4x8\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:39.523369 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.523369 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:39.523519 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:39.523379 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:40.211042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:40.210993 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" event={"ID":"6bef7ad0-3487-4e4c-a64e-7762eb4bdef4","Type":"ContainerDied","Data":"06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3"} Nov 01 22:42:40.211204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:40.211060 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06069b9b6701aa8a8335fd81e78bb604168d60e782ca4527ff62301819c48ba3" Nov 01 22:42:40.211204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:40.211007 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt" Nov 01 22:42:46.021869 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:46.021800 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv"] Nov 01 22:42:46.024387 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:46.024360 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2ppv"] Nov 01 22:42:47.163895 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:47.163862 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8462d1df-2ed7-48fa-a7f6-dc4ae19e035f" path="/var/lib/kubelet/pods/8462d1df-2ed7-48fa-a7f6-dc4ae19e035f/volumes" Nov 01 22:42:48.085331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.085290 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:42:48.085725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.085712 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" containerName="revoke-keys" Nov 01 22:42:48.085771 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.085726 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" containerName="revoke-keys" Nov 01 22:42:48.085823 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.085812 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" containerName="revoke-keys" Nov 01 22:42:48.089564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.089544 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.093549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.093530 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3813a1-openshift-default-dockercfg-phk6l\"" Nov 01 22:42:48.105559 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.105527 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:42:48.206216 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.206175 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.206739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.206234 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.206739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.206334 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.207187 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207159 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.207330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207215 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.207330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207260 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.207330 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207289 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9p2\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.207523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207359 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.208169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.207410 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309399 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309433 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309460 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9p2\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309541 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309627 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309662 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309711 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.309801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309741 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.310249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.310249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.309961 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.310249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.310186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.310249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.310201 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.310406 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.310384 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.312271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.312254 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.312352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.312336 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.322351 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.322327 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.322553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.322533 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9p2\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\") pod \"e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.402622 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.402535 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:48.685087 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.685058 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:42:48.687074 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:42:48.687016 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf277258d_a4d1_402b_9676_765d1457d725.slice/crio-6f4d517eaaaee69d6bc338609ea1a2dcd401aca239401fb1257c81aaad4171a3 WatchSource:0}: Error finding container 6f4d517eaaaee69d6bc338609ea1a2dcd401aca239401fb1257c81aaad4171a3: Status 404 returned error can't find the container with id 6f4d517eaaaee69d6bc338609ea1a2dcd401aca239401fb1257c81aaad4171a3 Nov 01 22:42:48.689463 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.689432 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:48.689554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.689496 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:48.689554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:48.689530 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:42:49.256952 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:49.256917 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerStarted","Data":"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b"} Nov 01 22:42:49.257421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:49.256966 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerStarted","Data":"6f4d517eaaaee69d6bc338609ea1a2dcd401aca239401fb1257c81aaad4171a3"} Nov 01 22:42:49.282356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:49.282311 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podStartSLOduration=1.282298909 podStartE2EDuration="1.282298909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:49.281007881 +0000 UTC m=+3516.254910417" watchObservedRunningTime="2026-11-01 22:42:49.282298909 +0000 UTC m=+3516.256201448" Nov 01 22:42:49.406774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:49.406625 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:49.417062 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:49.417001 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:50.262262 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:50.262237 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:50.263092 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:50.263073 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:42:52.301270 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:42:52.301231 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:42:52.750546 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.750518 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:42:52.754376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.754357 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.763749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.763721 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:42:52.764022 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.764001 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3813a1\"" Nov 01 22:42:52.854194 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.854156 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.854393 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.854327 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpms5\" (UniqueName: \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.955921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.955877 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpms5\" (UniqueName: \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.956162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.956018 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.956689 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.956669 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:52.968370 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:52.968350 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpms5\" (UniqueName: \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\") pod \"payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:53.076918 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:53.076850 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:53.361491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:53.361419 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:42:53.365012 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:42:53.364981 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a10839f_b6fc_417e_8843_96c89e3cb28e.slice/crio-20ac9e43621569f90e24ecbc37706241311d6a7bc711d3bc908334bf4fdd3ebc WatchSource:0}: Error finding container 20ac9e43621569f90e24ecbc37706241311d6a7bc711d3bc908334bf4fdd3ebc: Status 404 returned error can't find the container with id 20ac9e43621569f90e24ecbc37706241311d6a7bc711d3bc908334bf4fdd3ebc Nov 01 22:42:54.290648 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:54.290609 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" event={"ID":"5a10839f-b6fc-417e-8843-96c89e3cb28e","Type":"ContainerStarted","Data":"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0"} Nov 01 22:42:54.290858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:54.290672 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" event={"ID":"5a10839f-b6fc-417e-8843-96c89e3cb28e","Type":"ContainerStarted","Data":"20ac9e43621569f90e24ecbc37706241311d6a7bc711d3bc908334bf4fdd3ebc"} Nov 01 22:42:54.290858 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:54.290693 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:42:54.310050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:54.309977 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" podStartSLOduration=2.3099636119999998 podStartE2EDuration="2.309963612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:54.308777015 +0000 UTC m=+3521.282679554" watchObservedRunningTime="2026-11-01 22:42:54.309963612 +0000 UTC m=+3521.283866151" Nov 01 22:42:57.022470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:57.022434 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp"] Nov 01 22:42:57.025571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:57.025540 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g64zp"] Nov 01 22:42:57.163655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:42:57.163621 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2d83c4-9cec-41d5-8178-7cc0983882b4" path="/var/lib/kubelet/pods/2d2d83c4-9cec-41d5-8178-7cc0983882b4/volumes" Nov 01 22:43:05.264234 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.264197 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:05.264676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.264251 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:05.297979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.297952 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:43:05.367867 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.367838 4395 generic.go:356] "Generic (PLEG): container finished" podID="f277258d-a4d1-402b-9676-765d1457d725" containerID="ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b" exitCode=0 Nov 01 22:43:05.368052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.367929 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerDied","Data":"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b"} Nov 01 22:43:05.368470 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.368453 4395 scope.go:122] "RemoveContainer" containerID="ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b" Nov 01 22:43:05.369850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.369822 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:05.369950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.369889 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:05.369950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:05.369920 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 01 22:43:06.395997 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:06.395942 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerStarted","Data":"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916"} Nov 01 22:43:06.403065 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:06.403003 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:43:06.403384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:06.403362 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:06.403533 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:06.403513 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:07.403042 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:07.402996 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:07.403473 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:07.403112 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:08.403014 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:08.402973 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:08.403236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:08.403069 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:09.403304 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:09.403256 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:09.403759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:09.403319 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:10.403431 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:10.403396 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Nov 01 22:43:10.403866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:10.403458 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Nov 01 22:43:11.835297 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:11.835264 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:43:11.836270 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:11.836246 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:43:11.837162 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:11.837135 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:43:17.756341 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:17.756310 4395 scope.go:122] "RemoveContainer" containerID="fac654319a67a0e49aa8f7a1cf0fcfa233928bb938e606afdd9476d234155cb2" Nov 01 22:43:17.765312 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:17.765295 4395 scope.go:122] "RemoveContainer" containerID="a49298087d92a9c89b1e07df04a29eb552160154f450883d7be3393ae24426dd" Nov 01 22:43:17.773509 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:17.773492 4395 scope.go:122] "RemoveContainer" containerID="73c002ca769f15eafbb2756becf1efcd20994ef39c9ea212bf91195f01609f2b" Nov 01 22:43:17.781221 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:17.781193 4395 scope.go:122] "RemoveContainer" containerID="2fde8f48665a24e75104c3de5fb69e371a5e459c41ba543ec166e01ad830c7a9" Nov 01 22:43:22.305295 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:43:22.305254 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:43:38.522318 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.522277 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t"] Nov 01 22:43:38.526121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.526098 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.532850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.532825 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:43:38.532978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.532929 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:43:38.534490 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.534470 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t"] Nov 01 22:43:38.571609 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.571578 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.571765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.571740 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.571812 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.571764 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gbn\" (UniqueName: \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.672610 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.672584 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.672736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.672614 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gbn\" (UniqueName: \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.672736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.672646 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.673271 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.673252 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.675054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.675017 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.684946 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.684910 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gbn\" (UniqueName: \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\") pod \"maas-api-revoke-sub-e1001e15-9q26t\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:38.840459 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:38.840408 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:39.120712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:39.120689 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t"] Nov 01 22:43:39.122588 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:43:39.122561 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee62531_01a9_4ffe_9e16_8f98a17e8a51.slice/crio-4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d WatchSource:0}: Error finding container 4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d: Status 404 returned error can't find the container with id 4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d Nov 01 22:43:39.571001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:39.570968 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" event={"ID":"eee62531-01a9-4ffe-9e16-8f98a17e8a51","Type":"ContainerStarted","Data":"c0243f9ccd661ab8dbc53bc27d5c8cb53b08a5dd8e615bb10df63d7190762572"} Nov 01 22:43:39.571374 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:39.571037 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" event={"ID":"eee62531-01a9-4ffe-9e16-8f98a17e8a51","Type":"ContainerStarted","Data":"4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d"} Nov 01 22:43:39.591373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:39.591326 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" podStartSLOduration=1.591312463 podStartE2EDuration="1.591312463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:39.588943559 +0000 UTC m=+3566.562846125" watchObservedRunningTime="2026-11-01 22:43:39.591312463 +0000 UTC m=+3566.565215002" Nov 01 22:43:40.577835 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:40.577804 4395 generic.go:356] "Generic (PLEG): container finished" podID="eee62531-01a9-4ffe-9e16-8f98a17e8a51" containerID="c0243f9ccd661ab8dbc53bc27d5c8cb53b08a5dd8e615bb10df63d7190762572" exitCode=0 Nov 01 22:43:40.578265 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:40.577852 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" event={"ID":"eee62531-01a9-4ffe-9e16-8f98a17e8a51","Type":"ContainerDied","Data":"c0243f9ccd661ab8dbc53bc27d5c8cb53b08a5dd8e615bb10df63d7190762572"} Nov 01 22:43:41.725790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.725767 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:41.800116 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.800090 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\") pod \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " Nov 01 22:43:41.800254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.800165 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\" (UniqueName: \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\") pod \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " Nov 01 22:43:41.800254 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.800244 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\") pod \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\" (UID: \"eee62531-01a9-4ffe-9e16-8f98a17e8a51\") " Nov 01 22:43:41.800621 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.800593 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca" pod "eee62531-01a9-4ffe-9e16-8f98a17e8a51" (UID: "eee62531-01a9-4ffe-9e16-8f98a17e8a51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:41.802310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.802284 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca" pod "eee62531-01a9-4ffe-9e16-8f98a17e8a51" (UID: "eee62531-01a9-4ffe-9e16-8f98a17e8a51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:41.802417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.802333 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn" pod "eee62531-01a9-4ffe-9e16-8f98a17e8a51" (UID: "eee62531-01a9-4ffe-9e16-8f98a17e8a51"). InnerVolumeSpecName "kube-api-access-g6gbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:41.901874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.901847 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:41.901874 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.901871 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee62531-01a9-4ffe-9e16-8f98a17e8a51-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:41.902057 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:41.901880 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6gbn\" (UniqueName: \"kubernetes.io/projected/eee62531-01a9-4ffe-9e16-8f98a17e8a51-kube-api-access-g6gbn\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:42.590884 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:42.590850 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" event={"ID":"eee62531-01a9-4ffe-9e16-8f98a17e8a51","Type":"ContainerDied","Data":"4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d"} Nov 01 22:43:42.590884 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:42.590885 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t" Nov 01 22:43:42.591193 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:42.590922 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4851b3f77fa5bc225a47d90d69f12448b3d01c26344c0997d4d6227eb90f727d" Nov 01 22:43:49.359959 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.359924 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl"] Nov 01 22:43:49.360389 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.360374 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eee62531-01a9-4ffe-9e16-8f98a17e8a51" containerName="revoke-keys" Nov 01 22:43:49.360440 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.360390 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee62531-01a9-4ffe-9e16-8f98a17e8a51" containerName="revoke-keys" Nov 01 22:43:49.360491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.360481 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee62531-01a9-4ffe-9e16-8f98a17e8a51" containerName="revoke-keys" Nov 01 22:43:49.363080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.363060 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.370930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.370911 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:43:49.371071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.370950 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:43:49.372352 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.372316 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl"] Nov 01 22:43:49.489418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.489384 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.489593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.489429 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.489593 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.489565 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctv9\" (UniqueName: \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.590769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.590733 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctv9\" (UniqueName: \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.590938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.590803 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.590938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.590832 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.591427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.591405 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.593347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.593326 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.603141 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.603116 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctv9\" (UniqueName: \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.677977 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.677911 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:49.964361 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:49.964329 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl"] Nov 01 22:43:49.970056 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:43:49.969889 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31baf049_6859_45e6_a102_a7646bd33f9e.slice/crio-3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f WatchSource:0}: Error finding container 3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f: Status 404 returned error can't find the container with id 3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f Nov 01 22:43:50.632021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:50.631938 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" event={"ID":"31baf049-6859-45e6-a102-a7646bd33f9e","Type":"ContainerStarted","Data":"91ad0fd023e606d433fe8d898bddb16ae0b1ebf4773bad82b45c3e798b53edcc"} Nov 01 22:43:50.632021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:50.632002 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" event={"ID":"31baf049-6859-45e6-a102-a7646bd33f9e","Type":"ContainerStarted","Data":"3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f"} Nov 01 22:43:50.651642 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:50.651594 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" podStartSLOduration=1.6515804219999999 podStartE2EDuration="1.651580422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:50.650301668 +0000 UTC m=+3577.624204208" watchObservedRunningTime="2026-11-01 22:43:50.651580422 +0000 UTC m=+3577.625482961" Nov 01 22:43:51.638417 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:51.638383 4395 generic.go:356] "Generic (PLEG): container finished" podID="31baf049-6859-45e6-a102-a7646bd33f9e" containerID="91ad0fd023e606d433fe8d898bddb16ae0b1ebf4773bad82b45c3e798b53edcc" exitCode=0 Nov 01 22:43:51.638845 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:51.638474 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" event={"ID":"31baf049-6859-45e6-a102-a7646bd33f9e","Type":"ContainerDied","Data":"91ad0fd023e606d433fe8d898bddb16ae0b1ebf4773bad82b45c3e798b53edcc"} Nov 01 22:43:52.300944 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:43:52.300906 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:43:52.797284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.797256 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:52.923397 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.923305 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\" (UniqueName: \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\") pod \"31baf049-6859-45e6-a102-a7646bd33f9e\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " Nov 01 22:43:52.923567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.923399 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\") pod \"31baf049-6859-45e6-a102-a7646bd33f9e\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " Nov 01 22:43:52.923567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.923482 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\") pod \"31baf049-6859-45e6-a102-a7646bd33f9e\" (UID: \"31baf049-6859-45e6-a102-a7646bd33f9e\") " Nov 01 22:43:52.923747 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.923720 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca" pod "31baf049-6859-45e6-a102-a7646bd33f9e" (UID: "31baf049-6859-45e6-a102-a7646bd33f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:52.923906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.923886 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:52.925553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.925532 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca" pod "31baf049-6859-45e6-a102-a7646bd33f9e" (UID: "31baf049-6859-45e6-a102-a7646bd33f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:52.925643 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:52.925603 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9" pod "31baf049-6859-45e6-a102-a7646bd33f9e" (UID: "31baf049-6859-45e6-a102-a7646bd33f9e"). InnerVolumeSpecName "kube-api-access-bctv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:53.025358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:53.025330 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31baf049-6859-45e6-a102-a7646bd33f9e-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:53.025358 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:53.025356 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctv9\" (UniqueName: \"kubernetes.io/projected/31baf049-6859-45e6-a102-a7646bd33f9e-kube-api-access-bctv9\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:53.649503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:53.649467 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" Nov 01 22:43:53.649503 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:53.649480 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl" event={"ID":"31baf049-6859-45e6-a102-a7646bd33f9e","Type":"ContainerDied","Data":"3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f"} Nov 01 22:43:53.649765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:53.649530 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a1d31efcbb4731904c885d8f42473c7296bfa7e7684b2dbe44e09648d96781f" Nov 01 22:43:59.492749 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:43:59.492712 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:44:00.387504 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:00.387470 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl"] Nov 01 22:44:00.390383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:00.390354 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3813a1-j7tjl"] Nov 01 22:44:01.163294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:01.163262 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31baf049-6859-45e6-a102-a7646bd33f9e" path="/var/lib/kubelet/pods/31baf049-6859-45e6-a102-a7646bd33f9e/volumes" Nov 01 22:44:04.493743 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:04.493686 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" containerID="cri-o://196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0" gracePeriod=25 Nov 01 22:44:05.297082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:05.297019 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:07.059423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:07.059385 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:44:07.059947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:07.059675 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" containerID="cri-o://943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916" gracePeriod=30 Nov 01 22:44:11.839350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:11.839311 4395 patch_prober.go:29] interesting pod/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:11.839790 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:11.839381 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:12.328089 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.328063 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:44:12.505251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505228 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505263 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505293 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505335 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505361 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505384 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505417 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505475 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505504 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\") pod \"f277258d-a4d1-402b-9676-765d1457d725\" (UID: \"f277258d-a4d1-402b-9676-765d1457d725\") " Nov 01 22:44:12.505673 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505486 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:12.505907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505674 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:12.505907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505696 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:12.505907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505739 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:12.505907 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505816 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:12.506319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.505979 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f277258d-a4d1-402b-9676-765d1457d725-istiod-ca-cert\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.506319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.506000 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.506319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.506018 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-workload-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.506319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.506052 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-credential-socket\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.506319 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.506066 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-data\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.507529 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.507506 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:12.507656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.507632 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:12.507818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.507799 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:12.507883 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.507825 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2" pod "f277258d-a4d1-402b-9676-765d1457d725" (UID: "f277258d-a4d1-402b-9676-765d1457d725"). InnerVolumeSpecName "kube-api-access-sk9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:12.606553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.606528 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-istio-token\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.606553 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.606552 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9p2\" (UniqueName: \"kubernetes.io/projected/f277258d-a4d1-402b-9676-765d1457d725-kube-api-access-sk9p2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.606693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.606562 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f277258d-a4d1-402b-9676-765d1457d725-istio-podinfo\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.606693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.606570 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f277258d-a4d1-402b-9676-765d1457d725-istio-envoy\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.757043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.756966 4395 generic.go:356] "Generic (PLEG): container finished" podID="f277258d-a4d1-402b-9676-765d1457d725" containerID="943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916" exitCode=0 Nov 01 22:44:12.757043 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.757019 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" Nov 01 22:44:12.757224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.757076 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerDied","Data":"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916"} Nov 01 22:44:12.757224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.757143 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8" event={"ID":"f277258d-a4d1-402b-9676-765d1457d725","Type":"ContainerDied","Data":"6f4d517eaaaee69d6bc338609ea1a2dcd401aca239401fb1257c81aaad4171a3"} Nov 01 22:44:12.757224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.757174 4395 scope.go:122] "RemoveContainer" containerID="943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916" Nov 01 22:44:12.766383 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.766365 4395 scope.go:122] "RemoveContainer" containerID="ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b" Nov 01 22:44:12.774468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.774451 4395 scope.go:122] "RemoveContainer" containerID="943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916" Nov 01 22:44:12.774737 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:44:12.774721 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916\": container with ID starting with 943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916 not found: ID does not exist" containerID="943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916" Nov 01 22:44:12.774818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.774755 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916"} err="failed to get container status \"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916\": rpc error: code = NotFound desc = could not find container \"943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916\": container with ID starting with 943cfc635ca770f5348d5fac52abf8a27c44869be2d459774ea9957d5b892916 not found: ID does not exist" Nov 01 22:44:12.774818 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.774772 4395 scope.go:122] "RemoveContainer" containerID="ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b" Nov 01 22:44:12.775099 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:44:12.775064 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b\": container with ID starting with ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b not found: ID does not exist" containerID="ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b" Nov 01 22:44:12.775169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.775102 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b"} err="failed to get container status \"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b\": rpc error: code = NotFound desc = could not find container \"ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b\": container with ID starting with ff1df93ed436b1884330d469a95dfc663525258b81705fa7b319717143036b5b not found: ID does not exist" Nov 01 22:44:12.781764 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.781744 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:44:12.786399 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:12.786379 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3813a1-openshift-default-6477bb64c7mz2d8"] Nov 01 22:44:13.162769 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:13.162685 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f277258d-a4d1-402b-9676-765d1457d725" path="/var/lib/kubelet/pods/f277258d-a4d1-402b-9676-765d1457d725/volumes" Nov 01 22:44:15.297301 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:15.297259 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:16.907055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:16.905127 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:16.907055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:16.905203 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:17.309054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:17.307284 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:17.309054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:17.307354 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:17.812395 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:17.812361 4395 scope.go:122] "RemoveContainer" containerID="8d26ee82da235e97e57def95f43b021d6d145e278b17ed817b730d89532946f3" Nov 01 22:44:17.828097 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:17.828003 4395 scope.go:122] "RemoveContainer" containerID="6770b6522af9d89dd992bdfc878942c9d452057310f924338c3ae549289566bb" Nov 01 22:44:22.299656 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:44:22.299622 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:44:24.755994 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.755965 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:44:24.842146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.842106 4395 generic.go:356] "Generic (PLEG): container finished" podID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerID="196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0" exitCode=0 Nov 01 22:44:24.842277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.842148 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" Nov 01 22:44:24.842277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.842179 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" event={"ID":"5a10839f-b6fc-417e-8843-96c89e3cb28e","Type":"ContainerDied","Data":"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0"} Nov 01 22:44:24.842277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.842223 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp" event={"ID":"5a10839f-b6fc-417e-8843-96c89e3cb28e","Type":"ContainerDied","Data":"20ac9e43621569f90e24ecbc37706241311d6a7bc711d3bc908334bf4fdd3ebc"} Nov 01 22:44:24.842277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.842244 4395 scope.go:122] "RemoveContainer" containerID="196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0" Nov 01 22:44:24.850613 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.850598 4395 scope.go:122] "RemoveContainer" containerID="196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0" Nov 01 22:44:24.850890 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:44:24.850872 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0\": container with ID starting with 196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0 not found: ID does not exist" containerID="196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0" Nov 01 22:44:24.850956 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.850905 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0"} err="failed to get container status \"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0\": rpc error: code = NotFound desc = could not find container \"196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0\": container with ID starting with 196ec9216c289b2712f701d886f04a4aadcf4e13be057c6eaaf228f6e0f9e9b0 not found: ID does not exist" Nov 01 22:44:24.857231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.857212 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\") pod \"5a10839f-b6fc-417e-8843-96c89e3cb28e\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " Nov 01 22:44:24.857305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.857266 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\" (UniqueName: \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\") pod \"5a10839f-b6fc-417e-8843-96c89e3cb28e\" (UID: \"5a10839f-b6fc-417e-8843-96c89e3cb28e\") " Nov 01 22:44:24.857551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.857533 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume" pod "5a10839f-b6fc-417e-8843-96c89e3cb28e" (UID: "5a10839f-b6fc-417e-8843-96c89e3cb28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:24.859286 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.859269 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5" pod "5a10839f-b6fc-417e-8843-96c89e3cb28e" (UID: "5a10839f-b6fc-417e-8843-96c89e3cb28e"). InnerVolumeSpecName "kube-api-access-cpms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:24.958556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.958533 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a10839f-b6fc-417e-8843-96c89e3cb28e-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:24.958556 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:24.958555 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpms5\" (UniqueName: \"kubernetes.io/projected/5a10839f-b6fc-417e-8843-96c89e3cb28e-kube-api-access-cpms5\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:25.165325 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:25.165301 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:44:25.169372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:25.169353 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3813a1-74bd7tpzgp"] Nov 01 22:44:27.164435 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:27.164397 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" path="/var/lib/kubelet/pods/5a10839f-b6fc-417e-8843-96c89e3cb28e/volumes" Nov 01 22:44:31.901309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:31.901276 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8xkxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:31.902064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:31.902010 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8xkxn" podUID="65ac1800-ff60-4b0d-9967-5dc3a169c3d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:32.300597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:32.300556 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-hcdq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:32.319801 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:32.319759 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-hcdq8" podUID="d2d23e65-78fd-4b1d-ae55-0e12d154c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:44.573599 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.573566 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w"] Nov 01 22:44:44.574021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.573972 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.574021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.573984 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.574021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.573994 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.574021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574001 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.574021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574022 4395 cpu_manager.go:414] "containerMap: removing container" podUID="31baf049-6859-45e6-a102-a7646bd33f9e" containerName="revoke-keys" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574045 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31baf049-6859-45e6-a102-a7646bd33f9e" containerName="revoke-keys" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574055 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574060 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574115 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="31baf049-6859-45e6-a102-a7646bd33f9e" containerName="revoke-keys" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574124 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a10839f-b6fc-417e-8843-96c89e3cb28e" containerName="payload-pre-processing" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574133 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.574224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.574139 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f277258d-a4d1-402b-9676-765d1457d725" containerName="istio-proxy" Nov 01 22:44:44.577188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.577171 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.583711 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.583686 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:44.583988 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.583931 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:44:44.586445 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.586426 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w"] Nov 01 22:44:44.689687 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.689655 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcbr\" (UniqueName: \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.689807 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.689766 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.689892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.689879 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.791037 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.791007 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.791140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.791063 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcbr\" (UniqueName: \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.791140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.791123 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.791739 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.791721 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.793359 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.793343 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.803646 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.803623 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcbr\" (UniqueName: \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\") pod \"maas-api-revoke-sub-452b6d7b-r8n9w\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:44.891695 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:44.891650 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:45.172579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:45.172552 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w"] Nov 01 22:44:45.174255 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:44:45.174229 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d78981_663c_4d9d_bd3d_e0cbe2e464f6.slice/crio-1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983 WatchSource:0}: Error finding container 1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983: Status 404 returned error can't find the container with id 1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983 Nov 01 22:44:45.972909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:45.972350 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" event={"ID":"16d78981-663c-4d9d-bd3d-e0cbe2e464f6","Type":"ContainerStarted","Data":"dd41801530dc55fdabac1d245021d52ef2916041fe86bd9a0fbe8b1db4d97068"} Nov 01 22:44:45.972909 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:45.972421 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" event={"ID":"16d78981-663c-4d9d-bd3d-e0cbe2e464f6","Type":"ContainerStarted","Data":"1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983"} Nov 01 22:44:45.993724 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:45.993664 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" podStartSLOduration=1.9936446129999998 podStartE2EDuration="1.993644613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:45.989722163 +0000 UTC m=+3632.963624702" watchObservedRunningTime="2026-11-01 22:44:45.993644613 +0000 UTC m=+3632.967547153" Nov 01 22:44:46.983893 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:46.983857 4395 generic.go:356] "Generic (PLEG): container finished" podID="16d78981-663c-4d9d-bd3d-e0cbe2e464f6" containerID="dd41801530dc55fdabac1d245021d52ef2916041fe86bd9a0fbe8b1db4d97068" exitCode=0 Nov 01 22:44:46.984525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:46.984498 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" event={"ID":"16d78981-663c-4d9d-bd3d-e0cbe2e464f6","Type":"ContainerDied","Data":"dd41801530dc55fdabac1d245021d52ef2916041fe86bd9a0fbe8b1db4d97068"} Nov 01 22:44:48.660052 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.635351 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:48.860051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.859403 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\") pod \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " Nov 01 22:44:48.860051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.859473 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\" (UniqueName: \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\") pod \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " Nov 01 22:44:48.860051 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.859621 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\") pod \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\" (UID: \"16d78981-663c-4d9d-bd3d-e0cbe2e464f6\") " Nov 01 22:44:48.861259 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.861235 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca" pod "16d78981-663c-4d9d-bd3d-e0cbe2e464f6" (UID: "16d78981-663c-4d9d-bd3d-e0cbe2e464f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:48.865266 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.865242 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr" pod "16d78981-663c-4d9d-bd3d-e0cbe2e464f6" (UID: "16d78981-663c-4d9d-bd3d-e0cbe2e464f6"). InnerVolumeSpecName "kube-api-access-hbcbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:48.867181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.867154 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca" pod "16d78981-663c-4d9d-bd3d-e0cbe2e464f6" (UID: "16d78981-663c-4d9d-bd3d-e0cbe2e464f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:48.963815 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.963783 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:48.964093 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.964074 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcbr\" (UniqueName: \"kubernetes.io/projected/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-kube-api-access-hbcbr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:48.964251 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:48.964235 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d78981-663c-4d9d-bd3d-e0cbe2e464f6-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:49.029567 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:49.029526 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" event={"ID":"16d78981-663c-4d9d-bd3d-e0cbe2e464f6","Type":"ContainerDied","Data":"1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983"} Nov 01 22:44:49.029836 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:49.029816 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d18d24b11c8d3d13bf0a9f858a874b5c2b91daa7b7d2c4133aca8bf005b5983" Nov 01 22:44:49.029962 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:49.029868 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w" Nov 01 22:44:52.299911 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:44:52.299871 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:44:58.362111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.360500 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq"] Nov 01 22:44:58.362111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.361195 4395 cpu_manager.go:414] "containerMap: removing container" podUID="16d78981-663c-4d9d-bd3d-e0cbe2e464f6" containerName="revoke-keys" Nov 01 22:44:58.362111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.361216 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d78981-663c-4d9d-bd3d-e0cbe2e464f6" containerName="revoke-keys" Nov 01 22:44:58.362111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.361345 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d78981-663c-4d9d-bd3d-e0cbe2e464f6" containerName="revoke-keys" Nov 01 22:44:58.370654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.364999 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.370654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.365747 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq"] Nov 01 22:44:58.395587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.390681 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:58.461144 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.460582 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:44:58.495492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.493620 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.495492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.493747 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhbw\" (UniqueName: \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.495492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.493797 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.660153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.643190 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.660549 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.660517 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhbw\" (UniqueName: \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.660738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.660718 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.661748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.661722 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.669877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.669852 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.707071 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.707020 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhbw\" (UniqueName: \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\") pod \"maas-api-revoke-sub-3e31a506-vxzwq\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:58.807468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:58.807179 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:44:59.647053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:44:59.632995 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq"] Nov 01 22:44:59.668083 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:44:59.666447 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf72cab5_e93b_47e8_af8a_6607b6e12407.slice/crio-79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e WatchSource:0}: Error finding container 79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e: Status 404 returned error can't find the container with id 79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e Nov 01 22:45:00.138054 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:00.137858 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" event={"ID":"bf72cab5-e93b-47e8-af8a-6607b6e12407","Type":"ContainerStarted","Data":"79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e"} Nov 01 22:45:01.149654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:01.149549 4395 generic.go:356] "Generic (PLEG): container finished" podID="bf72cab5-e93b-47e8-af8a-6607b6e12407" containerID="ec8be2da5f9067b14b30bae3f6c59b0e900fff285efa39b85584fc1d8b632c1a" exitCode=0 Nov 01 22:45:01.150195 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:01.149641 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" event={"ID":"bf72cab5-e93b-47e8-af8a-6607b6e12407","Type":"ContainerDied","Data":"ec8be2da5f9067b14b30bae3f6c59b0e900fff285efa39b85584fc1d8b632c1a"} Nov 01 22:45:03.045200 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.034077 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:45:03.125164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.115229 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\" (UniqueName: \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\") pod \"bf72cab5-e93b-47e8-af8a-6607b6e12407\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " Nov 01 22:45:03.125164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.115299 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\") pod \"bf72cab5-e93b-47e8-af8a-6607b6e12407\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " Nov 01 22:45:03.173900 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.172433 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw" pod "bf72cab5-e93b-47e8-af8a-6607b6e12407" (UID: "bf72cab5-e93b-47e8-af8a-6607b6e12407"). InnerVolumeSpecName "kube-api-access-7zhbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:03.262714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.223974 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\") pod \"bf72cab5-e93b-47e8-af8a-6607b6e12407\" (UID: \"bf72cab5-e93b-47e8-af8a-6607b6e12407\") " Nov 01 22:45:03.263377 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.263345 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhbw\" (UniqueName: \"kubernetes.io/projected/bf72cab5-e93b-47e8-af8a-6607b6e12407-kube-api-access-7zhbw\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.263547 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.247263 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca" pod "bf72cab5-e93b-47e8-af8a-6607b6e12407" (UID: "bf72cab5-e93b-47e8-af8a-6607b6e12407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:03.295462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.277642 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca" pod "bf72cab5-e93b-47e8-af8a-6607b6e12407" (UID: "bf72cab5-e93b-47e8-af8a-6607b6e12407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:03.300053 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.296353 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" Nov 01 22:45:03.329525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.323010 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq" event={"ID":"bf72cab5-e93b-47e8-af8a-6607b6e12407","Type":"ContainerDied","Data":"79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e"} Nov 01 22:45:03.329964 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.329930 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e43dc8385638298568096e956ac26db4fb3a0c1a44e2ff36af81093aa83e0e" Nov 01 22:45:03.378912 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.378871 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.379222 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:03.379200 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf72cab5-e93b-47e8-af8a-6607b6e12407-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:07.022404 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:07.022363 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9"] Nov 01 22:45:07.025041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:07.025007 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b61e9352-w6dm9"] Nov 01 22:45:07.163501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:07.163466 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b26a34-e6c1-4d4a-81a1-7e2764a023b2" path="/var/lib/kubelet/pods/93b26a34-e6c1-4d4a-81a1-7e2764a023b2/volumes" Nov 01 22:45:11.519064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:11.518180 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cjw9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:11.519064 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:11.518263 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cjw9g" podUID="cdbb4f72-c168-4cba-ba1a-58b7eebc517f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:11.608055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:11.606121 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-gwqw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:11.608055 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:11.606199 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-gwqw2" podUID="071ca1b5-2ea4-4aef-8432-b6a745e3bd4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:17.871791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:17.871762 4395 scope.go:122] "RemoveContainer" containerID="1918c0ec14ed64dca5188ee8d098467889a2141b72f27dd1c27355e37273ee03" Nov 01 22:45:18.547939 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.547903 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp"] Nov 01 22:45:18.548338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.548324 4395 cpu_manager.go:414] "containerMap: removing container" podUID="bf72cab5-e93b-47e8-af8a-6607b6e12407" containerName="revoke-keys" Nov 01 22:45:18.548402 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.548339 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf72cab5-e93b-47e8-af8a-6607b6e12407" containerName="revoke-keys" Nov 01 22:45:18.548448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.548426 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf72cab5-e93b-47e8-af8a-6607b6e12407" containerName="revoke-keys" Nov 01 22:45:18.552791 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.552770 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.558366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.558162 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:18.558366 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.558202 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\"" Nov 01 22:45:18.559873 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.559849 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp"] Nov 01 22:45:18.620047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.619997 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmfr\" (UniqueName: \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.620181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.620135 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.620231 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.620188 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.721487 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.721458 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.721679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.721522 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.721679 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.721556 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmfr\" (UniqueName: \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.722214 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.722196 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.724016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.723984 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.733748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.733727 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmfr\" (UniqueName: \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\") pod \"maas-api-revoke-sub-74fd89f4-m6zmp\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:18.867926 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:18.867855 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:19.141727 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:19.141702 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp"] Nov 01 22:45:19.143513 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:45:19.143486 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0810e71_9823_42e4_bead_937f79644c6f.slice/crio-6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be WatchSource:0}: Error finding container 6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be: Status 404 returned error can't find the container with id 6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be Nov 01 22:45:19.433577 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:19.433546 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" event={"ID":"a0810e71-9823-42e4-bead-937f79644c6f","Type":"ContainerStarted","Data":"2063f5c713990bfabb8e6c58ee91c4d9fa582c5c58b35562d3dcd975756e1e19"} Nov 01 22:45:19.433693 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:19.433602 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" event={"ID":"a0810e71-9823-42e4-bead-937f79644c6f","Type":"ContainerStarted","Data":"6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be"} Nov 01 22:45:19.453298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:19.453254 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" podStartSLOduration=1.45324134 podStartE2EDuration="1.45324134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:19.450809539 +0000 UTC m=+3666.424712077" watchObservedRunningTime="2026-11-01 22:45:19.45324134 +0000 UTC m=+3666.427143878" Nov 01 22:45:20.440164 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:20.440129 4395 generic.go:356] "Generic (PLEG): container finished" podID="a0810e71-9823-42e4-bead-937f79644c6f" containerID="2063f5c713990bfabb8e6c58ee91c4d9fa582c5c58b35562d3dcd975756e1e19" exitCode=0 Nov 01 22:45:20.440558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:20.440179 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" event={"ID":"a0810e71-9823-42e4-bead-937f79644c6f","Type":"ContainerDied","Data":"2063f5c713990bfabb8e6c58ee91c4d9fa582c5c58b35562d3dcd975756e1e19"} Nov 01 22:45:21.582564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.582538 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:21.647249 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.647220 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\") pod \"a0810e71-9823-42e4-bead-937f79644c6f\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " Nov 01 22:45:21.647426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.647259 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\") pod \"a0810e71-9823-42e4-bead-937f79644c6f\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " Nov 01 22:45:21.647426 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.647342 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\" (UniqueName: \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\") pod \"a0810e71-9823-42e4-bead-937f79644c6f\" (UID: \"a0810e71-9823-42e4-bead-937f79644c6f\") " Nov 01 22:45:21.647683 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.647660 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca" pod "a0810e71-9823-42e4-bead-937f79644c6f" (UID: "a0810e71-9823-42e4-bead-937f79644c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:21.649513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.649489 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca" pod "a0810e71-9823-42e4-bead-937f79644c6f" (UID: "a0810e71-9823-42e4-bead-937f79644c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:21.649596 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.649514 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr" pod "a0810e71-9823-42e4-bead-937f79644c6f" (UID: "a0810e71-9823-42e4-bead-937f79644c6f"). InnerVolumeSpecName "kube-api-access-8tmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:21.748631 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.748568 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-xks-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.748631 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.748595 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0810e71-9823-42e4-bead-937f79644c6f-maas-api-openshift-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.748631 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:21.748605 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tmfr\" (UniqueName: \"kubernetes.io/projected/a0810e71-9823-42e4-bead-937f79644c6f-kube-api-access-8tmfr\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.300152 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:45:22.300119 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:45:22.451993 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:22.451956 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" Nov 01 22:45:22.452192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:22.452011 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp" event={"ID":"a0810e71-9823-42e4-bead-937f79644c6f","Type":"ContainerDied","Data":"6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be"} Nov 01 22:45:22.452192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:22.452084 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b566de25a5123ff64cd632cab897ada5f948269e866c3c9201b7a9541b695be" Nov 01 22:45:30.640478 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.640446 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:30.641073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.641054 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a0810e71-9823-42e4-bead-937f79644c6f" containerName="revoke-keys" Nov 01 22:45:30.641182 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.641075 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0810e71-9823-42e4-bead-937f79644c6f" containerName="revoke-keys" Nov 01 22:45:30.641240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.641220 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0810e71-9823-42e4-bead-937f79644c6f" containerName="revoke-keys" Nov 01 22:45:30.644554 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.644534 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.653748 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.653721 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:45:30.656896 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.656871 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:30.657111 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.657095 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 22:45:30.657203 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.657192 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zltrj\"" Nov 01 22:45:30.728914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.728875 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.729149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.729121 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.729224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.729176 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.729290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.729269 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99cq\" (UniqueName: \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.830290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.830258 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.830290 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.830291 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.830558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.830314 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99cq\" (UniqueName: \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.830558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.830380 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.830999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.830972 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.832810 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.832790 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.832887 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.832836 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.842861 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.842841 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99cq\" (UniqueName: \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\") pod \"maas-api-976bc5795-p2pgj\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:30.961227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:30.961149 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:31.241571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:31.241548 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:31.243390 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:45:31.243357 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16086922_9430_48ab_aa7d_8058cc711645.slice/crio-a20bb218648df17ad9daf2a8fc66a281a3430004e93f923395e3f153ad763fb5 WatchSource:0}: Error finding container a20bb218648df17ad9daf2a8fc66a281a3430004e93f923395e3f153ad763fb5: Status 404 returned error can't find the container with id a20bb218648df17ad9daf2a8fc66a281a3430004e93f923395e3f153ad763fb5 Nov 01 22:45:31.244742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:31.244723 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:45:31.506931 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:31.506894 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" event={"ID":"16086922-9430-48ab-aa7d-8058cc711645","Type":"ContainerStarted","Data":"a20bb218648df17ad9daf2a8fc66a281a3430004e93f923395e3f153ad763fb5"} Nov 01 22:45:32.513725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:32.513691 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" event={"ID":"16086922-9430-48ab-aa7d-8058cc711645","Type":"ContainerStarted","Data":"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135"} Nov 01 22:45:32.514149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:32.513760 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:32.535076 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:32.535010 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" podStartSLOduration=2.0618201369999998 podStartE2EDuration="2.534994544s" totalImagesPullingTime="473.174407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:32.533842029 +0000 UTC m=+3679.507744715" watchObservedRunningTime="2026-11-01 22:45:32.534994544 +0000 UTC m=+3679.508897130" Nov 01 22:45:37.023726 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:37.023676 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn"] Nov 01 22:45:37.025831 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:37.025809 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fdxzn"] Nov 01 22:45:37.163579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:37.163549 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18b196b-daee-4a2b-ab69-4da3d0f0e828" path="/var/lib/kubelet/pods/c18b196b-daee-4a2b-ab69-4da3d0f0e828/volumes" Nov 01 22:45:38.490212 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.490176 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:38.490742 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.490366 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" podUID="16086922-9430-48ab-aa7d-8058cc711645" containerName="maas-api" containerID="cri-o://6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135" gracePeriod=30 Nov 01 22:45:38.496139 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.496114 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:38.526552 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.526525 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4"] Nov 01 22:45:38.529910 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.529895 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.542903 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.542878 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4"] Nov 01 22:45:38.600474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.599828 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-metrics-certs\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.600474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.599887 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhln\" (UniqueName: \"kubernetes.io/projected/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-kube-api-access-9bhln\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.600474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.600198 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-openshift-service-ca\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.600474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.600268 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-maas-api-tls\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.701012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.700981 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-openshift-service-ca\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.701204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.701056 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-maas-api-tls\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.701204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.701127 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-metrics-certs\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.701204 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.701154 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhln\" (UniqueName: \"kubernetes.io/projected/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-kube-api-access-9bhln\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.701850 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.701823 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-openshift-service-ca\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.704059 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.704006 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-maas-api-tls\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.704169 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.704151 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-metrics-certs\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.714403 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.714380 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhln\" (UniqueName: \"kubernetes.io/projected/46af7ed7-bcd7-4e6f-9e98-6af8a565abf9-kube-api-access-9bhln\") pod \"maas-api-598d6b8b55-tkpq4\" (UID: \"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.747979 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.747935 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:38.801617 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.801591 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\") pod \"16086922-9430-48ab-aa7d-8058cc711645\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " Nov 01 22:45:38.801753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.801639 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\" (UniqueName: \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\") pod \"16086922-9430-48ab-aa7d-8058cc711645\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " Nov 01 22:45:38.801753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.801700 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\") pod \"16086922-9430-48ab-aa7d-8058cc711645\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " Nov 01 22:45:38.801753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.801737 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\") pod \"16086922-9430-48ab-aa7d-8058cc711645\" (UID: \"16086922-9430-48ab-aa7d-8058cc711645\") " Nov 01 22:45:38.802183 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.802157 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca" pod "16086922-9430-48ab-aa7d-8058cc711645" (UID: "16086922-9430-48ab-aa7d-8058cc711645"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:38.803943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.803915 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs" pod "16086922-9430-48ab-aa7d-8058cc711645" (UID: "16086922-9430-48ab-aa7d-8058cc711645"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:38.804418 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.804398 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls" pod "16086922-9430-48ab-aa7d-8058cc711645" (UID: "16086922-9430-48ab-aa7d-8058cc711645"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:38.804495 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.804417 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq" pod "16086922-9430-48ab-aa7d-8058cc711645" (UID: "16086922-9430-48ab-aa7d-8058cc711645"). InnerVolumeSpecName "kube-api-access-q99cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:38.875865 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.875841 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:38.903285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.903253 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99cq\" (UniqueName: \"kubernetes.io/projected/16086922-9430-48ab-aa7d-8058cc711645-kube-api-access-q99cq\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:38.903285 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.903287 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:38.903481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.903303 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16086922-9430-48ab-aa7d-8058cc711645-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:38.903481 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:38.903320 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16086922-9430-48ab-aa7d-8058cc711645-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:39.169384 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.169354 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4"] Nov 01 22:45:39.552651 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.552612 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" event={"ID":"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9","Type":"ContainerStarted","Data":"c15ec09402d3a5f9e3960dcfe96d6dd308f1755f16c28b48bd9491170f0c9719"} Nov 01 22:45:39.554041 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.553994 4395 generic.go:356] "Generic (PLEG): container finished" podID="16086922-9430-48ab-aa7d-8058cc711645" containerID="6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135" exitCode=0 Nov 01 22:45:39.554207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.554059 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" event={"ID":"16086922-9430-48ab-aa7d-8058cc711645","Type":"ContainerDied","Data":"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135"} Nov 01 22:45:39.554207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.554090 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" Nov 01 22:45:39.554207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.554110 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-p2pgj" event={"ID":"16086922-9430-48ab-aa7d-8058cc711645","Type":"ContainerDied","Data":"a20bb218648df17ad9daf2a8fc66a281a3430004e93f923395e3f153ad763fb5"} Nov 01 22:45:39.554207 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.554139 4395 scope.go:122] "RemoveContainer" containerID="6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135" Nov 01 22:45:39.563189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.563168 4395 scope.go:122] "RemoveContainer" containerID="6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135" Nov 01 22:45:39.563439 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:45:39.563420 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135\": container with ID starting with 6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135 not found: ID does not exist" containerID="6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135" Nov 01 22:45:39.563492 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.563446 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135"} err="failed to get container status \"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135\": rpc error: code = NotFound desc = could not find container \"6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135\": container with ID starting with 6b17e2d54f887b18ed358aae3e789ea21645cfceea4f908d495f8ee78e3a6135 not found: ID does not exist" Nov 01 22:45:39.571239 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.571216 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:39.575179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:39.575158 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-p2pgj"] Nov 01 22:45:40.561677 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:40.561644 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" event={"ID":"46af7ed7-bcd7-4e6f-9e98-6af8a565abf9","Type":"ContainerStarted","Data":"dc772727e41e779c3b0aad15acdd5aad5e2f625f54d6f7207d6d5bcb0061516b"} Nov 01 22:45:40.562493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:40.562474 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:40.585124 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:40.585075 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" podStartSLOduration=2.170937881 podStartE2EDuration="2.585061939s" totalImagesPullingTime="414.124058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:40.583565502 +0000 UTC m=+3687.557468043" watchObservedRunningTime="2026-11-01 22:45:40.585061939 +0000 UTC m=+3687.558964469" Nov 01 22:45:41.164834 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.164800 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16086922-9430-48ab-aa7d-8058cc711645" path="/var/lib/kubelet/pods/16086922-9430-48ab-aa7d-8058cc711645/volumes" Nov 01 22:45:41.378081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.378048 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:45:41.378566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.378547 4395 cpu_manager.go:414] "containerMap: removing container" podUID="16086922-9430-48ab-aa7d-8058cc711645" containerName="maas-api" Nov 01 22:45:41.378675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.378569 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16086922-9430-48ab-aa7d-8058cc711645" containerName="maas-api" Nov 01 22:45:41.378675 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.378657 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="16086922-9430-48ab-aa7d-8058cc711645" containerName="maas-api" Nov 01 22:45:41.382154 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.382131 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.391005 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.390969 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eaf797bf\"" Nov 01 22:45:41.392644 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.392616 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:45:41.415372 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.415314 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:45:41.418768 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.418745 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.432121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.430879 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:45:41.432121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.431822 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eaf797bf-dockercfg-kxnhb\"" Nov 01 22:45:41.448118 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.448086 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:45:41.452423 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.452400 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.464501 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.464476 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:45:41.527240 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527205 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.527385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527270 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.527385 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527339 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hxc\" (UniqueName: \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.527466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527394 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.527466 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527421 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.527576 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527535 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.527620 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527597 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km7p2\" (UniqueName: \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.527720 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.527703 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnrd\" (UniqueName: \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.530597 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.530576 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eaf797bf-serving-cert\"" Nov 01 22:45:41.628906 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.628877 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.628912 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.628972 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629023 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km7p2\" (UniqueName: \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629119 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnrd\" (UniqueName: \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629165 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629195 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.629356 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629270 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hxc\" (UniqueName: \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.629947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629842 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.629947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.629891 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.630336 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.630310 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.631740 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.631718 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.631833 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.631806 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.641754 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.641732 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hxc\" (UniqueName: \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\") pod \"maas-api-e2e-ait-eaf797bf-8c44d647b-tz828\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.641859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.641760 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km7p2\" (UniqueName: \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\") pod \"payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.642011 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.641994 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnrd\" (UniqueName: \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\") pod \"payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.697246 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.697089 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:41.734655 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.734623 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:41.771310 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.771242 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:41.985600 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:41.985575 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:45:41.987495 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:45:41.987467 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb814b435_ee1a_460a_8a9d_ad29410269fb.slice/crio-f1ad7d387771eb0c94e9307f4ccd892a59d6e53b1a0452fe48fea14f42ccb553 WatchSource:0}: Error finding container f1ad7d387771eb0c94e9307f4ccd892a59d6e53b1a0452fe48fea14f42ccb553: Status 404 returned error can't find the container with id f1ad7d387771eb0c94e9307f4ccd892a59d6e53b1a0452fe48fea14f42ccb553 Nov 01 22:45:42.021872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.021850 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:45:42.024354 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:45:42.024324 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5bba21_099f_4a88_9f7b_2ceb0fc8fdfa.slice/crio-85fac0e0a4f23cf42fa2ffbce447e442ab82bf5698c8494b10b2dc6b87eb9941 WatchSource:0}: Error finding container 85fac0e0a4f23cf42fa2ffbce447e442ab82bf5698c8494b10b2dc6b87eb9941: Status 404 returned error can't find the container with id 85fac0e0a4f23cf42fa2ffbce447e442ab82bf5698c8494b10b2dc6b87eb9941 Nov 01 22:45:42.075663 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.075638 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:45:42.077837 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:45:42.077810 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e94aeea_6fc1_4097_b914_1f4ec5ad60f6.slice/crio-9a2fbc74074db5b42cc0cba37922e4de57e19038cb548c94d9eb81a7e61d19ee WatchSource:0}: Error finding container 9a2fbc74074db5b42cc0cba37922e4de57e19038cb548c94d9eb81a7e61d19ee: Status 404 returned error can't find the container with id 9a2fbc74074db5b42cc0cba37922e4de57e19038cb548c94d9eb81a7e61d19ee Nov 01 22:45:42.580776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.580740 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" event={"ID":"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa","Type":"ContainerStarted","Data":"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9"} Nov 01 22:45:42.580947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.580795 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" event={"ID":"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa","Type":"ContainerStarted","Data":"85fac0e0a4f23cf42fa2ffbce447e442ab82bf5698c8494b10b2dc6b87eb9941"} Nov 01 22:45:42.580947 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.580832 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:42.582572 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.582546 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" event={"ID":"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6","Type":"ContainerStarted","Data":"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250"} Nov 01 22:45:42.582692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.582592 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" event={"ID":"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6","Type":"ContainerStarted","Data":"9a2fbc74074db5b42cc0cba37922e4de57e19038cb548c94d9eb81a7e61d19ee"} Nov 01 22:45:42.582692 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.582622 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:42.584219 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.584196 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" event={"ID":"b814b435-ee1a-460a-8a9d-ad29410269fb","Type":"ContainerStarted","Data":"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0"} Nov 01 22:45:42.584326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.584252 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" event={"ID":"b814b435-ee1a-460a-8a9d-ad29410269fb","Type":"ContainerStarted","Data":"f1ad7d387771eb0c94e9307f4ccd892a59d6e53b1a0452fe48fea14f42ccb553"} Nov 01 22:45:42.584326 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.584282 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:45:42.601728 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.601682 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" podStartSLOduration=1.601669961 podStartE2EDuration="1.601669961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:42.599850078 +0000 UTC m=+3689.573752616" watchObservedRunningTime="2026-11-01 22:45:42.601669961 +0000 UTC m=+3689.575572491" Nov 01 22:45:42.621165 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.621115 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" podStartSLOduration=1.234758972 podStartE2EDuration="1.621102779s" totalImagesPullingTime="386.343807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:42.619051206 +0000 UTC m=+3689.592953746" watchObservedRunningTime="2026-11-01 22:45:42.621102779 +0000 UTC m=+3689.595005318" Nov 01 22:45:42.638191 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:42.638149 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" podStartSLOduration=1.638139347 podStartE2EDuration="1.638139347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:42.636341407 +0000 UTC m=+3689.610243945" watchObservedRunningTime="2026-11-01 22:45:42.638139347 +0000 UTC m=+3689.612041913" Nov 01 22:45:46.575710 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:46.575682 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-tkpq4" Nov 01 22:45:48.595725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:48.595695 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:45:52.301781 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:45:52.301741 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:45:53.593965 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:53.593934 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:45:53.594465 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:45:53.594008 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:46:05.915797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:05.915764 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:46:05.930592 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:05.930560 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:46:06.259758 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.259723 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:46:06.259957 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.259934 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" podUID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" containerName="maas-api" containerID="cri-o://3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250" gracePeriod=30 Nov 01 22:46:06.548237 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.548212 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:46:06.621676 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.621645 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\") pod \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " Nov 01 22:46:06.621839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.621726 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\") pod \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " Nov 01 22:46:06.621839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.621772 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\" (UniqueName: \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\") pod \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " Nov 01 22:46:06.621983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.621909 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\") pod \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\" (UID: \"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6\") " Nov 01 22:46:06.622170 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.622144 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca" pod "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" (UID: "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:06.622309 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.622301 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-openshift-service-ca\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:06.624073 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.624048 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc" pod "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" (UID: "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6"). InnerVolumeSpecName "kube-api-access-x8hxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:06.624523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.624502 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs" pod "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" (UID: "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:06.624523 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.624508 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls" pod "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" (UID: "1e94aeea-6fc1-4097-b914-1f4ec5ad60f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:06.711607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.711573 4395 generic.go:356] "Generic (PLEG): container finished" podID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" containerID="3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250" exitCode=0 Nov 01 22:46:06.711796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.711637 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" Nov 01 22:46:06.711796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.711661 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" event={"ID":"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6","Type":"ContainerDied","Data":"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250"} Nov 01 22:46:06.711796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.711710 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828" event={"ID":"1e94aeea-6fc1-4097-b914-1f4ec5ad60f6","Type":"ContainerDied","Data":"9a2fbc74074db5b42cc0cba37922e4de57e19038cb548c94d9eb81a7e61d19ee"} Nov 01 22:46:06.711796 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.711730 4395 scope.go:122] "RemoveContainer" containerID="3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250" Nov 01 22:46:06.721694 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.721676 4395 scope.go:122] "RemoveContainer" containerID="3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250" Nov 01 22:46:06.721936 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:46:06.721917 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250\": container with ID starting with 3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250 not found: ID does not exist" containerID="3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250" Nov 01 22:46:06.721989 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.721943 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250"} err="failed to get container status \"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250\": rpc error: code = NotFound desc = could not find container \"3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250\": container with ID starting with 3a3a89d390b7291eb8c370a11fb253c425cca65939d61a0fb703bb020c474250 not found: ID does not exist" Nov 01 22:46:06.723759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.723737 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-metrics-certs\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:06.723759 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.723759 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-maas-api-tls\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:06.723936 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.723769 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hxc\" (UniqueName: \"kubernetes.io/projected/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6-kube-api-access-x8hxc\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:06.735287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.735263 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:46:06.739493 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:06.739469 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eaf797bf-8c44d647b-tz828"] Nov 01 22:46:07.163390 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:07.163359 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" path="/var/lib/kubelet/pods/1e94aeea-6fc1-4097-b914-1f4ec5ad60f6/volumes" Nov 01 22:46:10.916296 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:10.916178 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" podUID="b814b435-ee1a-460a-8a9d-ad29410269fb" containerName="payload-processing" containerID="cri-o://e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0" gracePeriod=25 Nov 01 22:46:10.931774 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:10.931736 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" podUID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" containerName="payload-pre-processing" containerID="cri-o://08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9" gracePeriod=25 Nov 01 22:46:11.200541 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.200516 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:46:11.243355 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.243332 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:46:11.271978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.271954 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\" (UniqueName: \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\") pod \"b814b435-ee1a-460a-8a9d-ad29410269fb\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " Nov 01 22:46:11.272104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.272089 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\") pod \"b814b435-ee1a-460a-8a9d-ad29410269fb\" (UID: \"b814b435-ee1a-460a-8a9d-ad29410269fb\") " Nov 01 22:46:11.272408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.272388 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume" pod "b814b435-ee1a-460a-8a9d-ad29410269fb" (UID: "b814b435-ee1a-460a-8a9d-ad29410269fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:11.274299 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.274278 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd" pod "b814b435-ee1a-460a-8a9d-ad29410269fb" (UID: "b814b435-ee1a-460a-8a9d-ad29410269fb"). InnerVolumeSpecName "kube-api-access-5hnrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:11.372733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.372708 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\" (UniqueName: \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\") pod \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " Nov 01 22:46:11.372853 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.372739 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\") pod \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\" (UID: \"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa\") " Nov 01 22:46:11.372928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.372914 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b814b435-ee1a-460a-8a9d-ad29410269fb-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:11.372967 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.372930 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnrd\" (UniqueName: \"kubernetes.io/projected/b814b435-ee1a-460a-8a9d-ad29410269fb-kube-api-access-5hnrd\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:11.373188 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.373169 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume" pod "ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" (UID: "ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:11.374934 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.374917 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2" pod "ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" (UID: "ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa"). InnerVolumeSpecName "kube-api-access-km7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:11.474379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.474318 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km7p2\" (UniqueName: \"kubernetes.io/projected/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-kube-api-access-km7p2\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:11.474379 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.474345 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:11.741879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.741808 4395 generic.go:356] "Generic (PLEG): container finished" podID="b814b435-ee1a-460a-8a9d-ad29410269fb" containerID="e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0" exitCode=0 Nov 01 22:46:11.741879 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.741856 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" Nov 01 22:46:11.742110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.741891 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" event={"ID":"b814b435-ee1a-460a-8a9d-ad29410269fb","Type":"ContainerDied","Data":"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0"} Nov 01 22:46:11.742110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.741945 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm" event={"ID":"b814b435-ee1a-460a-8a9d-ad29410269fb","Type":"ContainerDied","Data":"f1ad7d387771eb0c94e9307f4ccd892a59d6e53b1a0452fe48fea14f42ccb553"} Nov 01 22:46:11.742110 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.741970 4395 scope.go:122] "RemoveContainer" containerID="e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0" Nov 01 22:46:11.743561 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.743537 4395 generic.go:356] "Generic (PLEG): container finished" podID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" containerID="08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9" exitCode=0 Nov 01 22:46:11.743684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.743586 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" event={"ID":"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa","Type":"ContainerDied","Data":"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9"} Nov 01 22:46:11.743684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.743590 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" Nov 01 22:46:11.743684 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.743617 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv" event={"ID":"ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa","Type":"ContainerDied","Data":"85fac0e0a4f23cf42fa2ffbce447e442ab82bf5698c8494b10b2dc6b87eb9941"} Nov 01 22:46:11.757376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.757361 4395 scope.go:122] "RemoveContainer" containerID="e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0" Nov 01 22:46:11.757656 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:46:11.757636 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0\": container with ID starting with e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0 not found: ID does not exist" containerID="e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0" Nov 01 22:46:11.757763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.757663 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0"} err="failed to get container status \"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0\": rpc error: code = NotFound desc = could not find container \"e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0\": container with ID starting with e8a09631255ec79c529fed3d04466ceaf1af4569feab1bd769d71b2ce93b00d0 not found: ID does not exist" Nov 01 22:46:11.757763 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.757680 4395 scope.go:122] "RemoveContainer" containerID="08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9" Nov 01 22:46:11.766331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.766269 4395 scope.go:122] "RemoveContainer" containerID="08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9" Nov 01 22:46:11.766549 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:46:11.766521 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9\": container with ID starting with 08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9 not found: ID does not exist" containerID="08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9" Nov 01 22:46:11.766614 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.766559 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9"} err="failed to get container status \"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9\": rpc error: code = NotFound desc = could not find container \"08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9\": container with ID starting with 08e033ccf7781b09fa6e64de476647d09354bf526d80cd7645ea3eec5d1ee9a9 not found: ID does not exist" Nov 01 22:46:11.768448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.768427 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:46:11.772294 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.772274 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eaf797bf-747856577b-4h9sm"] Nov 01 22:46:11.784004 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.783982 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:46:11.790914 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:11.790894 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eaf797bf-588c684f59-pn9lv"] Nov 01 22:46:13.163179 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:13.163145 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b814b435-ee1a-460a-8a9d-ad29410269fb" path="/var/lib/kubelet/pods/b814b435-ee1a-460a-8a9d-ad29410269fb/volumes" Nov 01 22:46:13.163709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:13.163668 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" path="/var/lib/kubelet/pods/ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa/volumes" Nov 01 22:46:17.895210 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:17.895180 4395 scope.go:122] "RemoveContainer" containerID="210834bf6a7ada7f85ededa59abb47a8e07c3848bd8c1685cdcb64238db2d67e" Nov 01 22:46:17.904155 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:17.904060 4395 scope.go:122] "RemoveContainer" containerID="17948908f30ffb5c33ebf1be68789a059546a72607d87898447c16f2f1fcf1b3" Nov 01 22:46:18.134058 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134011 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:46:18.134658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134635 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" containerName="payload-pre-processing" Nov 01 22:46:18.134658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134658 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" containerName="payload-pre-processing" Nov 01 22:46:18.134856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134679 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b814b435-ee1a-460a-8a9d-ad29410269fb" containerName="payload-processing" Nov 01 22:46:18.134856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134687 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b814b435-ee1a-460a-8a9d-ad29410269fb" containerName="payload-processing" Nov 01 22:46:18.134856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134739 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" containerName="maas-api" Nov 01 22:46:18.134856 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134746 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" containerName="maas-api" Nov 01 22:46:18.135082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134882 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5bba21-099f-4a88-9f7b-2ceb0fc8fdfa" containerName="payload-pre-processing" Nov 01 22:46:18.135082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134898 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b814b435-ee1a-460a-8a9d-ad29410269fb" containerName="payload-processing" Nov 01 22:46:18.135082 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.134914 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e94aeea-6fc1-4097-b914-1f4ec5ad60f6" containerName="maas-api" Nov 01 22:46:18.139841 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.139822 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.147252 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.147178 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-1fef08\"" Nov 01 22:46:18.149558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.149537 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:46:18.164121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.164100 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-1fef08-dockercfg-t7z47\"" Nov 01 22:46:18.192298 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.192267 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:46:18.195866 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.195847 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.207960 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.207931 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:46:18.270885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.270776 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.271121 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.270976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4w7\" (UniqueName: \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.372313 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.372283 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.372474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.372367 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4w7\" (UniqueName: \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.372474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.372411 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.372474 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.372468 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjtv\" (UniqueName: \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.372995 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.372973 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.385392 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.385364 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4w7\" (UniqueName: \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\") pod \"payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.456570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.456404 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:18.474021 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.473996 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.474173 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.474093 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjtv\" (UniqueName: \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.474736 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.474716 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.486476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.486451 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjtv\" (UniqueName: \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\") pod \"payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.509429 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.509403 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:18.735761 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.735719 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:46:18.737489 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:46:18.737454 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358108ac_8f35_443e_a7a5_ed8452fb7c4f.slice/crio-c20697e09ff181a7d4765bc682a25ae559803722f394a72a1aa0ee3cbaa8ebc4 WatchSource:0}: Error finding container c20697e09ff181a7d4765bc682a25ae559803722f394a72a1aa0ee3cbaa8ebc4: Status 404 returned error can't find the container with id c20697e09ff181a7d4765bc682a25ae559803722f394a72a1aa0ee3cbaa8ebc4 Nov 01 22:46:18.792668 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.792639 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" event={"ID":"358108ac-8f35-443e-a7a5-ed8452fb7c4f","Type":"ContainerStarted","Data":"c20697e09ff181a7d4765bc682a25ae559803722f394a72a1aa0ee3cbaa8ebc4"} Nov 01 22:46:18.806691 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:18.806660 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:46:18.808067 ip-10-0-136-47 kubenswrapper[4395]: W1101 22:46:18.808017 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2455ecab_e106_44c7_801d_6638ee63f1c0.slice/crio-e39d0ac22dbd3db221b15017b4de4655cf5b90cce9c278424739277a380b38af WatchSource:0}: Error finding container e39d0ac22dbd3db221b15017b4de4655cf5b90cce9c278424739277a380b38af: Status 404 returned error can't find the container with id e39d0ac22dbd3db221b15017b4de4655cf5b90cce9c278424739277a380b38af Nov 01 22:46:19.798839 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.798793 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" event={"ID":"358108ac-8f35-443e-a7a5-ed8452fb7c4f","Type":"ContainerStarted","Data":"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9"} Nov 01 22:46:19.799333 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.798891 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:19.800452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.800430 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" event={"ID":"2455ecab-e106-44c7-801d-6638ee63f1c0","Type":"ContainerStarted","Data":"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f"} Nov 01 22:46:19.800575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.800467 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" event={"ID":"2455ecab-e106-44c7-801d-6638ee63f1c0","Type":"ContainerStarted","Data":"e39d0ac22dbd3db221b15017b4de4655cf5b90cce9c278424739277a380b38af"} Nov 01 22:46:19.800575 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.800520 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:19.820776 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.820732 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" podStartSLOduration=1.8207193670000001 podStartE2EDuration="1.820719367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:19.819671273 +0000 UTC m=+3726.793573813" watchObservedRunningTime="2026-11-01 22:46:19.820719367 +0000 UTC m=+3726.794621925" Nov 01 22:46:19.838985 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:19.838941 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" podStartSLOduration=1.8389299970000001 podStartE2EDuration="1.838929997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:19.837764977 +0000 UTC m=+3726.811667517" watchObservedRunningTime="2026-11-01 22:46:19.838929997 +0000 UTC m=+3726.812832535" Nov 01 22:46:22.300250 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:46:22.300218 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:46:30.808661 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:30.808568 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:46:30.809205 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:46:30.809182 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:46:52.302152 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:46:52.302110 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:47:17.946928 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:17.946895 4395 scope.go:122] "RemoveContainer" containerID="caf17897cabd673cdd3aeeed1934565d2c0f65ea2fef60026da9b0a88260fc1d" Nov 01 22:47:21.022498 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:21.022457 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc"] Nov 01 22:47:21.025189 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:21.025165 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xldbc"] Nov 01 22:47:21.173872 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:21.173835 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81d289e-f600-4317-bfc5-a97742ef67b8" path="/var/lib/kubelet/pods/a81d289e-f600-4317-bfc5-a97742ef67b8/volumes" Nov 01 22:47:22.301797 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:47:22.301761 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:47:40.020924 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:40.020880 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt"] Nov 01 22:47:40.028305 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:40.028274 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-k2wpt"] Nov 01 22:47:41.162632 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:47:41.162601 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bef7ad0-3487-4e4c-a64e-7762eb4bdef4" path="/var/lib/kubelet/pods/6bef7ad0-3487-4e4c-a64e-7762eb4bdef4/volumes" Nov 01 22:47:52.300791 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:47:52.300750 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:48:01.634779 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:01.634694 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:48:01.670382 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:01.670353 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:48:06.635767 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.635705 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" podUID="2455ecab-e106-44c7-801d-6638ee63f1c0" containerName="payload-processing" containerID="cri-o://b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f" gracePeriod=25 Nov 01 22:48:06.671119 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.671083 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" podUID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" containerName="payload-pre-processing" containerID="cri-o://0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9" gracePeriod=25 Nov 01 22:48:06.927264 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.927229 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:48:06.952974 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.952945 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:48:06.964046 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.964004 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\" (UniqueName: \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\") pod \"2455ecab-e106-44c7-801d-6638ee63f1c0\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " Nov 01 22:48:06.964215 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.964160 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\") pod \"2455ecab-e106-44c7-801d-6638ee63f1c0\" (UID: \"2455ecab-e106-44c7-801d-6638ee63f1c0\") " Nov 01 22:48:06.964551 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.964525 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume" pod "2455ecab-e106-44c7-801d-6638ee63f1c0" (UID: "2455ecab-e106-44c7-801d-6638ee63f1c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:06.966261 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:06.966233 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv" pod "2455ecab-e106-44c7-801d-6638ee63f1c0" (UID: "2455ecab-e106-44c7-801d-6638ee63f1c0"). InnerVolumeSpecName "kube-api-access-xgjtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:07.065277 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.065235 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\" (UniqueName: \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\") pod \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " Nov 01 22:48:07.065476 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.065289 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\") pod \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\" (UID: \"358108ac-8f35-443e-a7a5-ed8452fb7c4f\") " Nov 01 22:48:07.065524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.065484 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjtv\" (UniqueName: \"kubernetes.io/projected/2455ecab-e106-44c7-801d-6638ee63f1c0-kube-api-access-xgjtv\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:07.065524 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.065498 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2455ecab-e106-44c7-801d-6638ee63f1c0-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:07.065709 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.065677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume" pod "358108ac-8f35-443e-a7a5-ed8452fb7c4f" (UID: "358108ac-8f35-443e-a7a5-ed8452fb7c4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:07.067525 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.067502 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7" pod "358108ac-8f35-443e-a7a5-ed8452fb7c4f" (UID: "358108ac-8f35-443e-a7a5-ed8452fb7c4f"). InnerVolumeSpecName "kube-api-access-sz4w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:07.166416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.166334 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz4w7\" (UniqueName: \"kubernetes.io/projected/358108ac-8f35-443e-a7a5-ed8452fb7c4f-kube-api-access-sz4w7\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:07.166416 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.166370 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358108ac-8f35-443e-a7a5-ed8452fb7c4f-plugins-config-volume\") on node \"ip-10-0-136-47.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:07.365840 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.365807 4395 generic.go:356] "Generic (PLEG): container finished" podID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" containerID="0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9" exitCode=0 Nov 01 22:48:07.366081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.365850 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" Nov 01 22:48:07.366081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.365888 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" event={"ID":"358108ac-8f35-443e-a7a5-ed8452fb7c4f","Type":"ContainerDied","Data":"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9"} Nov 01 22:48:07.366081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.365942 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf" event={"ID":"358108ac-8f35-443e-a7a5-ed8452fb7c4f","Type":"ContainerDied","Data":"c20697e09ff181a7d4765bc682a25ae559803722f394a72a1aa0ee3cbaa8ebc4"} Nov 01 22:48:07.366081 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.365972 4395 scope.go:122] "RemoveContainer" containerID="0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9" Nov 01 22:48:07.367256 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.367233 4395 generic.go:356] "Generic (PLEG): container finished" podID="2455ecab-e106-44c7-801d-6638ee63f1c0" containerID="b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f" exitCode=0 Nov 01 22:48:07.367337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.367268 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" Nov 01 22:48:07.367337 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.367309 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" event={"ID":"2455ecab-e106-44c7-801d-6638ee63f1c0","Type":"ContainerDied","Data":"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f"} Nov 01 22:48:07.367460 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.367350 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99" event={"ID":"2455ecab-e106-44c7-801d-6638ee63f1c0","Type":"ContainerDied","Data":"e39d0ac22dbd3db221b15017b4de4655cf5b90cce9c278424739277a380b38af"} Nov 01 22:48:07.375566 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.375547 4395 scope.go:122] "RemoveContainer" containerID="0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9" Nov 01 22:48:07.375843 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:48:07.375823 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9\": container with ID starting with 0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9 not found: ID does not exist" containerID="0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9" Nov 01 22:48:07.375892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.375850 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9"} err="failed to get container status \"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9\": rpc error: code = NotFound desc = could not find container \"0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9\": container with ID starting with 0e0faf5479e70485baad9a84b813dc905b01359ea0d826a5b2cd555cb14c24f9 not found: ID does not exist" Nov 01 22:48:07.375892 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.375864 4395 scope.go:122] "RemoveContainer" containerID="b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f" Nov 01 22:48:07.384006 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.383987 4395 scope.go:122] "RemoveContainer" containerID="b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f" Nov 01 22:48:07.384294 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:48:07.384274 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f\": container with ID starting with b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f not found: ID does not exist" containerID="b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f" Nov 01 22:48:07.384376 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.384301 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f"} err="failed to get container status \"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f\": rpc error: code = NotFound desc = could not find container \"b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f\": container with ID starting with b8d9a274a322c44d4698e980d498145dcab88ffc18b8452fbcf952be98aa7b2f not found: ID does not exist" Nov 01 22:48:07.384978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.384958 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:48:07.389104 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.389079 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1fef08-686d85b69f-hh7bf"] Nov 01 22:48:07.400932 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.400909 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:48:07.405513 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:07.405492 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1fef08-7cb8479996-fpf99"] Nov 01 22:48:09.163192 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:09.163164 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2455ecab-e106-44c7-801d-6638ee63f1c0" path="/var/lib/kubelet/pods/2455ecab-e106-44c7-801d-6638ee63f1c0/volumes" Nov 01 22:48:09.163582 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:09.163504 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" path="/var/lib/kubelet/pods/358108ac-8f35-443e-a7a5-ed8452fb7c4f/volumes" Nov 01 22:48:22.313604 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:48:22.313557 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:48:42.033712 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:42.033172 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t"] Nov 01 22:48:42.035140 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:42.035118 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1001e15-9q26t"] Nov 01 22:48:43.164921 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:48:43.164879 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee62531-01a9-4ffe-9e16-8f98a17e8a51" path="/var/lib/kubelet/pods/eee62531-01a9-4ffe-9e16-8f98a17e8a51/volumes" Nov 01 22:48:52.301594 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:48:52.301543 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:49:13.546944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:13.546912 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5dd86d6db4-k28xn_f5d35462-db7a-4cd7-bd40-427a4be8d921/manager/0.log" Nov 01 22:49:13.729424 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:13.729389 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2c8m7_b03aef49-9d21-48e1-9f0f-71a95a6b5451/manager/0.log" Nov 01 22:49:14.323236 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:14.323205 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:49:14.531047 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:14.531000 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xgv6s_559bf8dc-f0e9-4069-aecc-b0d8de5e4111/server/0.log" Nov 01 22:49:15.830420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:15.830381 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-598d6b8b55-tkpq4_46af7ed7-bcd7-4e6f-9e98-6af8a565abf9/maas-api/0.log" Nov 01 22:49:16.357930 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:16.357907 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vxzwq_bf72cab5-e93b-47e8-af8a-6607b6e12407/revoke-keys/0.log" Nov 01 22:49:16.532050 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:16.531994 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-r8n9w_16d78981-663c-4d9d-bd3d-e0cbe2e464f6/revoke-keys/0.log" Nov 01 22:49:16.876067 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:16.876019 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-74fd89f4-m6zmp_a0810e71-9823-42e4-bead-937f79644c6f/revoke-keys/0.log" Nov 01 22:49:17.212345 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:17.212267 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-k248d_12a3a283-9b34-410d-80f7-77b82731ff51/postgres/0.log" Nov 01 22:49:18.039733 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:18.039702 4395 scope.go:122] "RemoveContainer" containerID="d9f30d3b6a3bf99a32e4d790fe43687e486725ed95ded113912caa80bf4e6636" Nov 01 22:49:18.048457 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:18.048437 4395 scope.go:122] "RemoveContainer" containerID="f35dc2a538211fc522b17f31cf88803c3dbe088c9932bd381789fcecc87debf8" Nov 01 22:49:19.188538 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:19.188512 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4qfv4_7b221c69-1f05-4a39-8186-66503ad4055c/kuadrant-console-plugin/0.log" Nov 01 22:49:20.642018 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:20.641985 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v64b4_f7db28f1-af5a-43b6-a5b9-68183ebbb8b4/discovery/0.log" Nov 01 22:49:20.800587 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:20.800559 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-9rkd4_aca37cfc-3c06-4afc-a12d-20cd8600f5a9/kube-auth-proxy/0.log" Nov 01 22:49:21.137227 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:21.137193 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8xkxn_65ac1800-ff60-4b0d-9967-5dc3a169c3d8/istio-proxy/0.log" Nov 01 22:49:21.301468 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:21.301438 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cjw9g_cdbb4f72-c168-4cba-ba1a-58b7eebc517f/istio-proxy/0.log" Nov 01 22:49:21.645752 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:21.645713 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gwqw2_071ca1b5-2ea4-4aef-8432-b6a745e3bd4a/istio-proxy/0.log" Nov 01 22:49:21.819580 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:21.819549 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcdq8_d2d23e65-78fd-4b1d-ae55-0e12d154c846/istio-proxy/0.log" Nov 01 22:49:22.301008 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:49:22.300972 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:49:23.038814 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:23.038784 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dxpk7_b48ace94-f380-4e38-917f-8f266e8ab437/payload-pre-processing/0.log" Nov 01 22:49:23.197816 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:23.197792 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:49:24.082009 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:24.081974 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk_1429c5b6-feda-4fa4-8fd9-f041fe7f98ef/main/0.log" Nov 01 22:49:24.257452 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:24.257426 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-svbr7_79984a24-8785-4acc-8259-cc1e4f908fb5/main/0.log" Nov 01 22:49:24.430999 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:24.430892 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2mhpk_3febf414-14ee-4545-abfa-1f421ad43f31/main/0.log" Nov 01 22:49:25.104278 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:25.104248 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64_a389fbce-17d4-4613-884d-37aa85377992/main/0.log" Nov 01 22:49:34.670943 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:34.670912 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5dd86d6db4-k28xn_f5d35462-db7a-4cd7-bd40-427a4be8d921/manager/0.log" Nov 01 22:49:34.861421 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:34.861391 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2c8m7_b03aef49-9d21-48e1-9f0f-71a95a6b5451/manager/0.log" Nov 01 22:49:35.419080 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:35.419052 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:49:35.631084 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:35.631058 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xgv6s_559bf8dc-f0e9-4069-aecc-b0d8de5e4111/server/0.log" Nov 01 22:49:36.529950 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:36.529909 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-598d6b8b55-tkpq4_46af7ed7-bcd7-4e6f-9e98-6af8a565abf9/maas-api/0.log" Nov 01 22:49:37.068448 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:37.068419 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vxzwq_bf72cab5-e93b-47e8-af8a-6607b6e12407/revoke-keys/0.log" Nov 01 22:49:37.233340 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:37.233308 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-r8n9w_16d78981-663c-4d9d-bd3d-e0cbe2e464f6/revoke-keys/0.log" Nov 01 22:49:37.574990 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:37.574962 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-74fd89f4-m6zmp_a0810e71-9823-42e4-bead-937f79644c6f/revoke-keys/0.log" Nov 01 22:49:37.925434 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:37.925358 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-k248d_12a3a283-9b34-410d-80f7-77b82731ff51/postgres/0.log" Nov 01 22:49:39.536953 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:39.536925 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4qfv4_7b221c69-1f05-4a39-8186-66503ad4055c/kuadrant-console-plugin/0.log" Nov 01 22:49:40.778941 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:40.778909 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v64b4_f7db28f1-af5a-43b6-a5b9-68183ebbb8b4/discovery/0.log" Nov 01 22:49:40.949153 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:40.949124 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-9rkd4_aca37cfc-3c06-4afc-a12d-20cd8600f5a9/kube-auth-proxy/0.log" Nov 01 22:49:41.288146 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:41.288115 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8xkxn_65ac1800-ff60-4b0d-9967-5dc3a169c3d8/istio-proxy/0.log" Nov 01 22:49:41.454471 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:41.454446 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cjw9g_cdbb4f72-c168-4cba-ba1a-58b7eebc517f/istio-proxy/0.log" Nov 01 22:49:41.797412 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:41.797390 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gwqw2_071ca1b5-2ea4-4aef-8432-b6a745e3bd4a/istio-proxy/0.log" Nov 01 22:49:41.972472 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:41.972446 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcdq8_d2d23e65-78fd-4b1d-ae55-0e12d154c846/istio-proxy/0.log" Nov 01 22:49:43.145208 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:43.145175 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dxpk7_b48ace94-f380-4e38-917f-8f266e8ab437/payload-pre-processing/0.log" Nov 01 22:49:43.313338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:43.313311 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:49:44.048570 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:44.048541 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-l5tpk_1429c5b6-feda-4fa4-8fd9-f041fe7f98ef/main/0.log" Nov 01 22:49:44.226780 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:44.226742 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-svbr7_79984a24-8785-4acc-8259-cc1e4f908fb5/main/0.log" Nov 01 22:49:44.395224 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:44.395130 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2mhpk_3febf414-14ee-4545-abfa-1f421ad43f31/main/0.log" Nov 01 22:49:45.063350 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:45.063319 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cgj64_a389fbce-17d4-4613-884d-37aa85377992/main/0.log" Nov 01 22:49:49.024569 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:49.024537 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w"] Nov 01 22:49:49.027438 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:49.027414 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-r8n9w"] Nov 01 22:49:49.163558 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:49:49.163527 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d78981-663c-4d9d-bd3d-e0cbe2e464f6" path="/var/lib/kubelet/pods/16d78981-663c-4d9d-bd3d-e0cbe2e464f6/volumes" Nov 01 22:49:52.300593 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:49:52.300557 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:50:04.021287 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:04.021238 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq"] Nov 01 22:50:04.023775 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:04.023749 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxzwq"] Nov 01 22:50:05.163491 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:05.163453 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf72cab5-e93b-47e8-af8a-6607b6e12407" path="/var/lib/kubelet/pods/bf72cab5-e93b-47e8-af8a-6607b6e12407/volumes" Nov 01 22:50:18.071458 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:18.071423 4395 scope.go:122] "RemoveContainer" containerID="c0243f9ccd661ab8dbc53bc27d5c8cb53b08a5dd8e615bb10df63d7190762572" Nov 01 22:50:18.080098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:18.080079 4395 scope.go:122] "RemoveContainer" containerID="91ad0fd023e606d433fe8d898bddb16ae0b1ebf4773bad82b45c3e798b53edcc" Nov 01 22:50:19.905938 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:19.905902 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:50:20.142528 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:20.142504 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:50:20.787462 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:20.787430 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:50:20.951061 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:20.951011 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:50:22.020975 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:22.020910 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp"] Nov 01 22:50:22.023555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:22.023531 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74fd89f4-m6zmp"] Nov 01 22:50:22.301122 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:50:22.301019 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:50:23.163347 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:50:23.163315 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0810e71-9823-42e4-bead-937f79644c6f" path="/var/lib/kubelet/pods/a0810e71-9823-42e4-bead-937f79644c6f/volumes" Nov 01 22:50:52.299771 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:50:52.299735 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:51:03.583890 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:03.583809 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:51:03.820685 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:03.820654 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56cb88789f-pxh6p_99ab3f18-0a51-4d73-a8b7-ff9866dbeb2a/manager/0.log" Nov 01 22:51:04.503855 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:04.503826 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:51:04.703338 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:04.703313 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:51:08.315016 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:08.314987 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4w9sx_6eaffc4a-a597-4820-8a11-31e92606f9f2/global-pull-secret-syncer/0.log" Nov 01 22:51:08.665944 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:08.665860 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7wgdj_5ff79f3e-c818-4103-b626-3bed7d7b46b4/konnectivity-agent/0.log" Nov 01 22:51:08.822555 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:08.822498 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-47.us-east-2.compute.internal_91729725e20792f4990b48a3f50d1312/haproxy/0.log" Nov 01 22:51:16.513658 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:16.513628 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4qfv4_7b221c69-1f05-4a39-8186-66503ad4055c/kuadrant-console-plugin/0.log" Nov 01 22:51:17.970714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.970672 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g"] Nov 01 22:51:17.971314 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971294 4395 cpu_manager.go:414] "containerMap: removing container" podUID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" containerName="payload-pre-processing" Nov 01 22:51:17.971420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971315 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" containerName="payload-pre-processing" Nov 01 22:51:17.971420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971344 4395 cpu_manager.go:414] "containerMap: removing container" podUID="2455ecab-e106-44c7-801d-6638ee63f1c0" containerName="payload-processing" Nov 01 22:51:17.971420 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971353 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2455ecab-e106-44c7-801d-6638ee63f1c0" containerName="payload-processing" Nov 01 22:51:17.971615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971508 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="2455ecab-e106-44c7-801d-6638ee63f1c0" containerName="payload-processing" Nov 01 22:51:17.971615 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.971534 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="358108ac-8f35-443e-a7a5-ed8452fb7c4f" containerName="payload-pre-processing" Nov 01 22:51:17.974719 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.974698 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:17.982762 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.982732 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"openshift-service-ca.crt\"" Nov 01 22:51:17.982917 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.982903 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"kube-root-ca.crt\"" Nov 01 22:51:17.982978 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.982961 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gx6ks\"/\"default-dockercfg-8xcsn\"" Nov 01 22:51:17.983738 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:17.983714 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g"] Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103443 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppsq\" (UniqueName: \"kubernetes.io/projected/25f64f9c-3bc1-4b80-84f3-3749b54792b7-kube-api-access-zppsq\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103567 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-sys\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103624 4395 scope.go:122] "RemoveContainer" containerID="ec8be2da5f9067b14b30bae3f6c59b0e900fff285efa39b85584fc1d8b632c1a" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103732 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-proc\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103778 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-podres\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.103869 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-lib-modules\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.110863 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.104752 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48klz_3e55adf1-4ef3-49bf-a410-e80a919374f7/node-exporter/0.log" Nov 01 22:51:18.119427 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.119408 4395 scope.go:122] "RemoveContainer" containerID="dd41801530dc55fdabac1d245021d52ef2916041fe86bd9a0fbe8b1db4d97068" Nov 01 22:51:18.129440 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.129419 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48klz_3e55adf1-4ef3-49bf-a410-e80a919374f7/kube-rbac-proxy/0.log" Nov 01 22:51:18.153725 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.153707 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48klz_3e55adf1-4ef3-49bf-a410-e80a919374f7/init-textfile/0.log" Nov 01 22:51:18.204765 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204744 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-lib-modules\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.204877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204798 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppsq\" (UniqueName: \"kubernetes.io/projected/25f64f9c-3bc1-4b80-84f3-3749b54792b7-kube-api-access-zppsq\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.204877 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204846 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-sys\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204883 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-proc\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204887 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-lib-modules\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204937 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-podres\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204947 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-sys\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205001 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204973 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-podres\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.205226 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.204941 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25f64f9c-3bc1-4b80-84f3-3749b54792b7-proc\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.218532 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.218507 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppsq\" (UniqueName: \"kubernetes.io/projected/25f64f9c-3bc1-4b80-84f3-3749b54792b7-kube-api-access-zppsq\") pod \"perf-node-gather-daemonset-r726g\" (UID: \"25f64f9c-3bc1-4b80-84f3-3749b54792b7\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.290334 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.290308 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:18.564951 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.564927 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g"] Nov 01 22:51:18.658098 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.658076 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hl584_71383c47-4a77-4bc3-9f3f-2e4cecdbe99f/prometheus-operator/0.log" Nov 01 22:51:18.678859 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.678843 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hl584_71383c47-4a77-4bc3-9f3f-2e4cecdbe99f/kube-rbac-proxy/0.log" Nov 01 22:51:18.709656 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.709609 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nbjcp_14d7ca00-d8c0-4916-b9f3-1fe77186a182/prometheus-operator-admission-webhook/0.log" Nov 01 22:51:18.740970 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.740948 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c64ccbbb-m8ln4_d8428012-b58b-4b52-ac62-85cce3271c5d/telemeter-client/0.log" Nov 01 22:51:18.765983 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.765963 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c64ccbbb-m8ln4_d8428012-b58b-4b52-ac62-85cce3271c5d/reload/0.log" Nov 01 22:51:18.791181 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:18.791162 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67c64ccbbb-m8ln4_d8428012-b58b-4b52-ac62-85cce3271c5d/kube-rbac-proxy/0.log" Nov 01 22:51:19.401103 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:19.401071 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" event={"ID":"25f64f9c-3bc1-4b80-84f3-3749b54792b7","Type":"ContainerStarted","Data":"c8bece8f04a61609d156fa7beff9d8d3e2df355c9d3e476ea1432a00bae8b40e"} Nov 01 22:51:19.401521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:19.401121 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" event={"ID":"25f64f9c-3bc1-4b80-84f3-3749b54792b7","Type":"ContainerStarted","Data":"a8bc21f32a1263fbcc29e66e823b06bbfde9b25647b5da2ad1287da6876aa0be"} Nov 01 22:51:19.401521 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:19.401266 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:19.420223 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:19.420178 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" podStartSLOduration=2.4201654599999998 podStartE2EDuration="2.42016546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:19.418620594 +0000 UTC m=+4026.392523136" watchObservedRunningTime="2026-11-01 22:51:19.42016546 +0000 UTC m=+4026.394067999" Nov 01 22:51:21.203571 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:21.203545 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7486c6f8cc-7jrh4_c2298c97-73a2-4ef7-8cea-8f2aad5190c8/console/0.log" Nov 01 22:51:21.241446 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:21.241420 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-9fvgv_78773b68-9492-41e9-948b-832944834e9e/download-server/0.log" Nov 01 22:51:22.300768 ip-10-0-136-47 kubenswrapper[4395]: E1101 22:51:22.300733 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4vbj" containerName="extractor" Nov 01 22:51:23.230607 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:23.230577 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rqr8c_6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4/dns/0.log" Nov 01 22:51:23.254564 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:23.254545 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rqr8c_6ca5f5bc-7dfd-4b19-ae0c-2d654538e1b4/kube-rbac-proxy/0.log" Nov 01 22:51:23.308230 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:23.308211 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mc686_eb2b9e93-478d-4b5f-bb3d-301f8b0c8be0/dns-node-resolver/0.log" Nov 01 22:51:24.023654 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:24.023617 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b7d4c5ff6-hs6vv_a1e5511f-afdf-4afd-850e-ba9171fcc326/registry/0.log" Nov 01 22:51:24.048331 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:24.048303 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbcqn_4ea9a939-d8f8-4d26-b7eb-57e2b88ba5c5/node-ca/0.log" Nov 01 22:51:25.416579 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:25.416550 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-r726g" Nov 01 22:51:27.672422 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:27.672392 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v64b4_f7db28f1-af5a-43b6-a5b9-68183ebbb8b4/discovery/0.log" Nov 01 22:51:28.115149 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.115119 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-9rkd4_aca37cfc-3c06-4afc-a12d-20cd8600f5a9/kube-auth-proxy/0.log" Nov 01 22:51:28.185012 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.184982 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8xkxn_65ac1800-ff60-4b0d-9967-5dc3a169c3d8/istio-proxy/0.log" Nov 01 22:51:28.222885 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.222865 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cjw9g_cdbb4f72-c168-4cba-ba1a-58b7eebc517f/istio-proxy/0.log" Nov 01 22:51:28.307269 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.307244 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gwqw2_071ca1b5-2ea4-4aef-8432-b6a745e3bd4a/istio-proxy/0.log" Nov 01 22:51:28.344699 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.344671 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcdq8_d2d23e65-78fd-4b1d-ae55-0e12d154c846/istio-proxy/0.log" Nov 01 22:51:28.564063 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.564039 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dxpk7_b48ace94-f380-4e38-917f-8f266e8ab437/payload-pre-processing/0.log" Nov 01 22:51:28.591284 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:28.591263 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:51:31.385753 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.385713 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-v64b4_f7db28f1-af5a-43b6-a5b9-68183ebbb8b4/discovery/0.log" Nov 01 22:51:31.395888 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.395869 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-9rkd4_aca37cfc-3c06-4afc-a12d-20cd8600f5a9/kube-auth-proxy/0.log" Nov 01 22:51:31.429797 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.429775 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8xkxn_65ac1800-ff60-4b0d-9967-5dc3a169c3d8/istio-proxy/0.log" Nov 01 22:51:31.447241 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.447217 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cjw9g_cdbb4f72-c168-4cba-ba1a-58b7eebc517f/istio-proxy/0.log" Nov 01 22:51:31.481373 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.481346 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-gwqw2_071ca1b5-2ea4-4aef-8432-b6a745e3bd4a/istio-proxy/0.log" Nov 01 22:51:31.501408 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.501384 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-hcdq8_d2d23e65-78fd-4b1d-ae55-0e12d154c846/istio-proxy/0.log" Nov 01 22:51:31.628657 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.628624 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dxpk7_b48ace94-f380-4e38-917f-8f266e8ab437/payload-pre-processing/0.log" Nov 01 22:51:31.643245 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.643181 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-tfddh_3db02aed-580e-42bc-88f7-9b7b06b6ce3b/payload-processing/0.log" Nov 01 22:51:31.650281 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:31.650256 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bj7ms_bb80e501-2a27-4977-ae29-3804022b6d13/serve-healthcheck-canary/0.log" Nov 01 22:51:32.440039 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:32.440005 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4vbj_d60f2807-1f57-4879-91c6-21d9b47c3917/kube-rbac-proxy/0.log" Nov 01 22:51:32.463349 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:32.463320 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4vbj_d60f2807-1f57-4879-91c6-21d9b47c3917/exporter/0.log" Nov 01 22:51:32.488328 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:32.488308 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4vbj_d60f2807-1f57-4879-91c6-21d9b47c3917/extractor/0.log" Nov 01 22:51:35.226714 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:35.226667 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-ncl5k_6bd474d8-3c94-4873-929a-f0caa9eba09a/manager/0.log" Nov 01 22:51:35.247220 ip-10-0-136-47 kubenswrapper[4395]: I1101 22:51:35.247198 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-ncl5k_6bd474d8-3c94-4873-929a-f0caa9eba09a/copy-manifests/0.log"